Discourse AI - NSFW

Discourse · April 24, 2023, 7:40pm

This topic covers the configuration of the NSFW (Not Safe For Work) feature of the Discourse AI plugin.

Required user level: Administrator

The NSFW modules can automatically classify the NSFW score of every new image upload in posts and chat messages in your Discourse instance. This system helps identify and manage content that may be inappropriate or explicit for a professional or public environment.

You can also enable automatic flagging of content that crosses a threshold.

Settings

ai_nsfw_detection_enabled: Enables or disables the module
ai_nsfw_inference_service_api_endpoint: URL where the API is running for the module. If you are using CDCK hosting this is automatically handled for you. If you are self-hosting check the self-hosting guide.
ai_nsfw_inference_service_api_key: API key for the API configured above. If you are using CDCK hosting this is automatically handled for you. If you are self-hosting check the self-hosting guide.
ai_nsfw_models: The model we’ll use for image classification. We offer two different ones with their thresholds:
- opennsfw2 returns a single score between 0.0 and 1.0. Set the threshold for an upload to be considered NSFW through the ai_nsfw_flag_threshold_general setting.
- nsfw_detector returns scores for four categories: drawings, hentai, porn, and sexy. Set the threshold for each one through the respective ai_nsfw_flag_threshold_${category} setting. If the general one is lower than any of these, we’ll consider the content NSFW.
ai_nsfw_flag_automatically: Automatically flag posts/chat messages that are above the configured thresholds. When this setting is disabled, we only store the classification results in the database.

Additional resources

Last edited by @hugh 2024-08-06T05:37:13Z

Last checked by @hugh 2024-08-06T05:45:45Z

Check document
Perform check on document:

nodomain · April 29, 2023, 10:19pm

Thanks.

I found this issue when using the nsfw service, after uploading an image. I use S3 for image storage, if this is relevant.

INFO:     10.0.2.145:23412 - "POST /api/v1/classify HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 77, in call_next
    message = await recv_stream.receive()
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.9/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/dist-packages/starlette_exporter/middleware.py", line 289, in __call__
    await self.app(scope, receive, wrapped_send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 106, in __call__
    response = await self.dispatch_func(request, call_next)
  File "./app.py", line 49, in dispatch
    response = await call_next(request)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 80, in call_next
    raise app_exc
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 69, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "./app.py", line 79, in classify
    results = models[parsed_params.model](tfile)
  File "/usr/local/lib/python3.9/dist-packages/opennsfw2/_inference.py", line 31, in predict_image
    pil_image = Image.open(image_path)
  File "/usr/local/lib/python3.9/dist-packages/PIL/Image.py", line 3283, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file '/tmp/tmpbqq82655'
ERROR:uvicorn.error:Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 77, in call_next
    message = await recv_stream.receive()
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.9/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/dist-packages/starlette_exporter/middleware.py", line 289, in __call__
    await self.app(scope, receive, wrapped_send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 106, in __call__
    response = await self.dispatch_func(request, call_next)
  File "./app.py", line 49, in dispatch
    response = await call_next(request)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 80, in call_next
    raise app_exc
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/base.py", line 69, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.9/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "./app.py", line 79, in classify
    results = models[parsed_params.model](tfile)
  File "/usr/local/lib/python3.9/dist-packages/opennsfw2/_inference.py", line 31, in predict_image
    pil_image = Image.open(image_path)
  File "/usr/local/lib/python3.9/dist-packages/PIL/Image.py", line 3283, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file '/tmp/tmpbqq82655'

In Sidekiq it looks like this:

What am I doing wrong?

nodomain · April 30, 2023, 8:36am

I think this maybe related to S3 really. When uploading a picture, the error logs shows these errors:
URL to S3 redacted…

So it seems to be that the plugin tries to load the image too early or it does not find it due to other reasons.

Falco · April 30, 2023, 7:48pm

I only tested it with sites using a CDN for S3 or using local storage. If using Object Storage I advise using a CDN to front it.

nodomain · April 30, 2023, 9:36pm

That’s exactly what we do - AWS Cloudfront. The stack is based on GitHub - aws-samples/aws-cdk-for-discourse: AWS CDK for Discourse. I’ll try to debug and post my results.

Noxitb · May 20, 2024, 3:41am

Where do i get the api key for it from

NateDhaliwal · June 6, 2024, 9:41am

It should be under the Admin panel, no?

Shauny · July 15, 2024, 12:23pm

This seems to be way too sensitive for me, it flags things far too often. Just pictures of people get flagged as NSFW.

I have the thresholds at 60. It’s not clear if I should increase this number or decrease it, and what the limit is? If I set it to 100 is that essentially disabling it? There doesn’t seem to be much documentation about it.

Also, why is there no way to completely disable it for certain roles, like there is with toxicity? I’m pretty certain our staff members won’t post anything rude, yet they are constantly fighting with the AI because it flags their stuff. It’s become a bit of a joke on my forum.

Using opennsfw2.

Falco · August 1, 2024, 4:02pm

Unfortunately those older ML models lack flexibility and while they are easy on the requirements for running your own instance, we found that the strictness was too much of a roadblock for the diverse set of communities using Discourse.

Currently our plan is to deprecate all the features relying on those old models and migrate everything to modern LLMs, where you will be able to tweak the prompt to better fit your needs.

Shauny · October 6, 2024, 4:08pm

Has there been any change in this at all? The NSFW detector is still woefully inaccurate, and triggers on most pictures with people in.

Falco · October 6, 2024, 5:26pm

Yes, we have communicated our plan at What’s next for NSFW detection in Discourse AI, but the TL;DR is that you can use Discourse AI Post Classifier - Automation rule for much better accuracy and flexibility.

We’ve ran evals for the NSFW use case via Vision LLMs and got great results, so we are currently rolling this out for our Enterprise customers and will publish our guides on this in the comings weeks.

Topic		Replies	Views
Discourse AI - AI triage Site Management automation , how-to , ai	26	2876	October 2, 2024
Discourse AI - Toxicity Site Management ai , ai-toxicity	7	2938	August 26, 2024
Discourse 3.1 is here! Blog	3	1257	January 31, 2024
Building a technical support chatbot Extras chat , ai	1	277	October 4, 2024
Secure Uploads Announcements secure-uploads	44	15829	October 25, 2024

Discourse AI - NSFW

Settings

Additional resources

Related topics