nsfwspy / nsfwspy.net Goto Github PK

View Code? Open in Web Editor NEW

313.0 8.0 22.0 203.04 MB

A .NET image and video classifier used to identify explicit/pornographic content written in C#.

License: MIT License

C# 71.22% HTML 4.16% SCSS 3.48% TypeScript 21.14%

nsfw porn pornography nudity nude inappropriate image-classification detector machine-learning nsfw-recognition

nsfwspy.net's People

Contributors

Stargazers

Watchers

nsfwspy.net's Issues

Hentai when all labels are 0

Hey, there's a minor issue like this, when i try to classify some "tough for AI neutral images":

here's the image used for this call:

here's my piece of code using your NuGet:

public NsfwSpyResult Classify(IFormFile file)
    {
        var nsfwSpy = new NsfwSpy();
        using (var ms = new MemoryStream())
        {
            file.CopyTo(ms);
            var fileBytes = ms.ToArray();
            var result = nsfwSpy.ClassifyImage(fileBytes);
            
            return result;
        }
        
    }

Dataset

Thank you for this repo. Is there any possibility to get access to the dataset used for training the network?

Incorrect nsfw detection

Apparently it detects YouTube links as nsfw? And pretty much any imgur or tenor link

Training

Hi,

Thanks for sharing this, it's pretty useful.

Could you advise how I can add additional classification?
I would like to add and classify gambling related images.

Also I've tried to run the Train project but getting an error on following line;

var trainedModel = trainingPipeline.Fit(trainSet);

And the error message is;

System.ArgumentOutOfRangeException: 'Only one class found in the LabelAsKey column. To build a multiclass classification model, the number of classes needs to be 2 or greater Arg_ParamName_Name'

Many thanks

Console application automatically crashes while classifying the image

Im running a discord bot on railway.app, The bot automatically disconnects and reconnects while classifying the image.

var nsfwImagedetector = new NsfwSpy(); var uri = new Uri(message.Attachments.First().Url); var result = nsfwImagedetector.ClassifyImage(uri); float resultFloat = result.Sexy + result.Hentai + result.Pornography; bool isImageNsfw = resultFloat > 0.992000 ? true : false; Console.WriteLine(resultFloat); Console.WriteLine(isImageNsfw); if (isImageNsfw) { var nsfwUpdate = Database.createUpdateSet("nsfw", true); await cardCollection.UpdateOneAsync(filter, nsfwUpdate); }

These are some errors i found while classifying image. although it gives result regardless of errors. Only issue is crashing the console application
`2023-01-07 13:07:39.898091: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2023-01-07 13:07:39.921942: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2023-01-07 13:07:40.576038: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-07 13:07:40.728653: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2200000000 Hz

2023-01-07 13:07:40.733302: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbea0005c30 initialized for platform Host (this does not guarantee that XLA will be used). Devices:

2023-01-07 13:07:40.733356: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version

2023-01-07 13:07:40.851229: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory

2023-01-07 13:07:40.851288: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)

2023-01-07 13:07:40.851336: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (railway): /proc/driver/nvidia/version does not exist

Have any plan about implement python version support?

Image classification fails on M1 chip Macs

Thanks for a great work. I just tried to add this package to my test C# API and failed to run it on MacOS Monterey.

After successfully adding to our project, when the code hit NSFW Spy, it exits immediately without any error. Any tips would be appreciated.

Installed SciSharp.TensorFlow.Redist
Injected dependency
Environment:
.NET SDK
Version: 6.0.105
Commit: 1c35735293

Runtime Environment:
OS Name: Mac OS X
OS Version: 12.4
OS Platform: Darwin
RID: osx.12-x64
Base Path: /usr/local/share/dotnet/x64/sdk/6.0.105/

Host (useful for support):
Version: 6.0.5
Commit: 70ae3df4a6

.NET SDKs installed:
3.1.419 [/usr/local/share/dotnet/x64/sdk]
5.0.408 [/usr/local/share/dotnet/x64/sdk]
6.0.105 [/usr/local/share/dotnet/x64/sdk]

Looking up the MIME of a file fails on Linux

On my system, I have nVidia Drivers installed, but no nVidia GPU at the moment. It crashes when it finds the driver but not apparently some magic files.

2022-05-27 22:50:07.900705: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nix/store/gka59hya7l7qp26s0rydcgq8hj0d7v7k-gcc-11.3.0-lib/lib:/nix/store/p9d5i8nhx5g77h6mjfs8rqfl2cq33jrr-libsecret-0.20.5/lib:/nix/store/9757vpby0rqzlbm2p02dm812h28djrg5-e2fsprogs-1.46.5/lib:/nix/store/l55cmvhfrqyxhdvgcfxgq0jpizcdp7ld-libnotify-0.7.12/lib:/nix/store/m7d2jrrwprb4n48ddccsyg2ffqr6yj0m-pipewire-0.3.51-jack/lib
2022-05-27 22:50:07.900735: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-05-27 22:50:08.072930: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-27 22:50:08.091265: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3593110000 Hz
2022-05-27 22:50:08.091881: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f92340093e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-05-27 22:50:08.091913: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2022-05-27 22:50:08.093389: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nix/store/gka59hya7l7qp26s0rydcgq8hj0d7v7k-gcc-11.3.0-lib/lib:/nix/store/p9d5i8nhx5g77h6mjfs8rqfl2cq33jrr-libsecret-0.20.5/lib:/nix/store/9757vpby0rqzlbm2p02dm812h28djrg5-e2fsprogs-1.46.5/lib:/nix/store/l55cmvhfrqyxhdvgcfxgq0jpizcdp7ld-libnotify-0.7.12/lib:/nix/store/m7d2jrrwprb4n48ddccsyg2ffqr6yj0m-pipewire-0.3.51-jack/lib
2022-05-27 22:50:08.093408: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2022-05-27 22:50:08.093431: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (uGamingPC): /proc/driver/nvidia/version does not exist
Unhandled exception. HeyRed.Mime.MagicException: Could not find any valid magic files!
   at HeyRed.Mime.Magic..ctor(MagicOpenFlags flags, String dbPath)
   at HeyRed.Mime.MimeGuesser.GuessMimeType(Byte[] buffer)
   at HeyRed.Mime.MimeGuesser.GuessFileType(Byte[] buffer)
   at NsfwSpyNS.NsfwSpy.ClassifyImage(Byte[] imageData)
   at NsfwSpyNS.NsfwSpy.ClassifyImage(Uri uri, WebClient webClient)
   at IsItYiff.Program.Main(String[] args) in /home/krutonium/RiderProjects/IsItYiff/IsItYiff/Program.cs:line 11

Predict Take a long time

Thanks for your great library but i use this library in web api project and it take a long time and at the end it cant predict anything. i run your test code and its not work.

new NsfwSpy() cause program to abort with no error message or exception

Trying to write a small quick command line program to scan a disk for NSFW images and report what it finds, all very simple stuff using .NET 6.

"Directory.GetFiles" to get a list of file names to check, then new up an NsfwSpy object, then a foreach to classify each found image in the list.

Problem is, when the program gets to the "var nsfwSpy = new NsfwSpy();" line, it pauses for a few seconds then just exits.

I've single stepped in as far as I can and it seems to be in the file NsfwSpy.cs at around line 28 where it creates a new MLContext with the model zip file.

The model is present, all the required DLL's are present, no errors, no exceptions or anything are given, just a silent exit and the program stops executing at that point.

Win10 64 bit, developing using JetBrains rider with .NET V6 and NsfwSpy added using NuGet (Version 3.4.3)

Any advice?

what type of this project model?

I want to transfer it to tflite model , use it in android project,thanks

ClassifyVideo does not work and other issues

Created a dumb console app (.NET 6) with the following
static void Main(string[] args)
{

        Console.WriteLine("Hello, World!");
        var nsfwSpy = new NsfwSpy();
      
        var result = nsfwSpy.ClassifyVideo(@"D:\movies for checking\Edit.mp4");

       
        Console.ReadKey();
    }

It seems that the code execution never comes out of using (var collection = new MagickImageCollection(video, MagickFormat.Mp4))
No exceptions or anything, i even left it running overnight thinking it might be taking long (the Edit.mp4 is first 10 minute clip of God Father Part 2), I cut it only to 10 min thinking it might be the size/time of all the movie being at (3h:22m run time and 1.4GB).
Here is what my console looks like as the program is running.

I have a 3070 running, with all the libraries installed properly (I think)

Hello, World!
2023-11-29 09:49:57.528567: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2023-11-29 09:49:57.727815: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-29 09:49:57.739943: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2602ddd4620 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-11-29 09:49:57.740026: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2023-11-29 09:49:57.741758: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2023-11-29 09:49:57.758938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2023-11-29 09:49:57.759034: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2023-11-29 09:49:58.152471: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2023-11-29 09:49:58.200779: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2023-11-29 09:49:58.228869: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2023-11-29 09:49:58.447674: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2023-11-29 09:49:58.677289: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2023-11-29 09:49:58.928462: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
>>>>>2023-11-29 09:49:58.928755:** I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2023-11-29 09:51:52.426382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
>>>>>>2023-11-29 09:51:52.426450:** I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2023-11-29 09:51:52.426524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2023-11-29 09:51:52.426887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6591 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:01:00.0, compute capability: 8.6)
2023-11-29 09:51:52.429524: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2606633c450 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-11-29 09:51:52.429562: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 3070, Compute Capability 8.6

I think u should be upgrading to .NET 6 at least (preferably 8 as 2.0 is deprecated), also note the time in bold in the above, it took a while for it to get to var collection = new MagickImageCollection(video, MagickFormat.Mp4)

Seems like the SciSharp, Magick, and Microsoft.ML can also be upgraded to later versions

I also wanted to do a PR where I add a progress event to the video processing so one can know how its going, below is the code if u wanna incorporate.

        //Code Added
        public delegate void UpdateVideoProgressDelegate(int currentFrame, int totalFrames);
        public event UpdateVideoProgressDelegate UpdateVideoProgress;
        
        public NsfwSpyFramesResult ClassifyVideo(byte[] video, VideoOptions videoOptions = null)
        {
            if (videoOptions == null)
                videoOptions = new VideoOptions();

            if (videoOptions.ClassifyEveryNthFrame < 1)
                throw new Exception("VideoOptions.ClassifyEveryNthFrame must not be less than 1.");

            var results = new ConcurrentDictionary<int, NsfwSpyResult>();

            var watch = Stopwatch.StartNew();
            using (var collection = new MagickImageCollection(video, MagickFormat.Mp4))
            {
                Console.WriteLine($"Loading video took {watch.Elapsed} to load video into collection of images");
                collection.Coalesce();
                var frameCount = collection.Count;

                Parallel.For(0, frameCount, (i, state) =>
                {
                    if (i % videoOptions.ClassifyEveryNthFrame != 0)
                        return;

                    if (state.ShouldExitCurrentIteration)
                        return;

                    var frame = collection[i];
                    frame.Format = MagickFormat.Jpg;

                    var result = ClassifyImage(frame.ToByteArray());
                    results.GetOrAdd(i, result);

                    //Code Added
                    if (videoOptions.EmitProgress)
                    {
                        UpdateVideoProgress?.Invoke(i, frameCount);
                    }

                    // Stop classifying frames if Nsfw frame is found
                    if (result.IsNsfw && videoOptions.EarlyStopOnNsfw)
                        state.Break();
                });
            }

            var resultDictionary = results.OrderBy(r => r.Key).ToDictionary(r => r.Key, r => r.Value);
            var gifResult = new NsfwSpyFramesResult(resultDictionary);
            return gifResult;
        }

Also how long does it take to process a 10 min vid in general.

NsfwSpy.App Incompatibility with Node.js v18.6.0+ / v16.17.0+

When attempting to build the NsfwSpy web app when using v18.6.0+ / v16.17.0+ of Node.js, it will throw out a "ERR_LOADER_CHAIN_INCOMPLETE" error and get stuck trying to run 'npm install' on loop. Node.js v16.16.0 works perfectly fine due to "ERR_LOADER_CHAIN_INCOMPLETE" not existing as an error in previous versions.

Classify multiple images , best way to remove lamda?

I've used NsfwSpy successfully on a few test files , I'm now trying to set it up where the classification is written to a Dictionary.

Based on the results I want to run a function that moves the file into a corresponding directory based on the classification.

The issue I'm having currently is because I don't want to mess with the nuget package, all the images in a directory and classified and then the Dictionary is ready and files are moved. But if there is an error during classification the program crashes and the sorting function isn't used.

Question is , what is the best way to break up the lamda function so that I can sort files based on the classification? It will make it quiet a bit faster too so I can sort each file as it's classified instead of writing the entire list , then reading the entire list to sort.

Net Framework 4.6.2

Good thing, you could create compatibility with net framework 4.6.2

Is there a .exe/GUI planned?

Straight up question I believe, but I'm interested in knowing if there are plans for a 'public' build availability

DI Scope in Web APIs

Hi there, I would like to host NsfwSpy inside a simple WebAPI controller to determine whether or not incoming images are NSFW. My question - if I register NsfwSpy in the DI system as Scoped, will new instances also consume resources? I.e. if one instance consumes ~600mb RAM, will 10 simultaneous incoming classification requests require ~6gb of RAM? Alternatively, can I safely register NsfwSpy as a DI Singleton without encountering threading issues or similar? What do you recommend as a best practice for this situation? Note - I am happy processing all incoming requests sequentially, my workload does not require parallel classification at this stage.

And thank you very much for providing this package!