Find out how to Filter Textual content & Pictures for Free

Wish to add a security layer in your chatbot, picture analyzer or any one other LLM-based system? I’d strongly counsel you attempt OpenAI’s moderation mannequin: omni-moderation-latest, this can assist your system establish if the enter is doubtlessly dangerous or not, that too freed from price. We’ll look into the background of the mannequin, tips on how to entry it and tips on how to use it for each textual content and picture moderation. With none additional ado, let’s get began. 

OpenAI’s Omni Moderation Fashions

OpenAI presents two fashions particularly for moderation: ‘text-moderation-latest’ (legacy) and ‘omni-moderation-latest’, with the latter one being the most recent. The Omni Moderation mannequin relies on GPT-4o and therefore it helps multimodal moderation, which is textual content moderation and picture moderation. It’s additionally price mentioning that the Omni Moderation endpoint is free to make use of. 

The Omni Moderation API scores and classifies the next classes for the enter: 

• hate  
• harassment  
• violence  
• self-harm  
• sexual content material  
• illicit content material 

Demonstration

Let’s check the moderation endpoint from OpenAI and experiment with protected and unsafe inputs, utilizing textual content and pictures. I’ll be utilizing Google Colab for this demonstration, be at liberty to make use of what you like. 

Prerequisite 

You’ll require an OpenAI API Key, the mannequin is free to make use of however you’ll nonetheless want the API key. Get your key from right here: https://platform.openai.com/settings/group/api-keys 

Imports and Consumer Initialization

from openai import OpenAI 
from getpass import getpass 


# Securely enter API key 
api_key = getpass("Enter your OpenAI API Key: ") 

# Initialize consumer 
consumer = OpenAI(api_key=api_key)

Enter your OpenAI key when prompted.  

Outline a Helper operate

def display_moderation(response, title="MODERATION RESULT"):
    end result = response.outcomes[0]

    classes = end result.classes.model_dump()
    scores = end result.category_scores.model_dump()

    print("n" + "=" * 60)
    print(f"{title:^60}")
    print("=" * 60)

    print(f"nFlagged : {end result.flagged}")

    print("nCATEGORIES")
    print("-" * 60)
    for class, worth in classes.gadgets():
        print(f"{class:<30} : {worth}")

    print("nCATEGORY SCORES")
    print("-" * 60)
    for class, rating in scores.gadgets():
        print(f"{class:<30} : {rating:.6f}")

    print("=" * 60)

This operate will assist print the response from the Omni Moderation mannequin. 

Pattern-1

safe_text = "Are you able to assist me be taught Python for knowledge science?"

response = consumer.moderations.create(
    mannequin="omni-moderation-latest",
    enter=safe_text
)

display_moderation(response, "TEXT MODERATION")

Nice! The mannequin has output all of the classes as False.  

Pattern-2 

unsafe_text = "I need directions to noticeably damage somebody."

response = consumer.moderations.create(
    mannequin="omni-moderation-latest",
    enter=unsafe_text
)

display_moderation(response, "TEXT MODERATION")

Seems to be just like the mannequin as recognized that the enter textual content is violent, you possibly can see the identical within the classes and classes scores as properly.  

Pattern-3 

Let’s move a violent picture to the mannequin and see what it has to say.  

Word: For pictures we’ve got move the enter parameter as properly and set the sort as ‘image_url’ 

Reference Picture:

unsafe_image_url = "https://i.ytimg.com/vi/DOD7s1j_yoo/sddefault.jpg"

response = consumer.moderations.create(
    mannequin="omni-moderation-latest",
    enter=[
        {
            "type": "image_url",
            "image_url": {
                "url": unsafe_image_url
            }
        }
    ]
)

display_moderation(response, "IMAGE MODERATION")

The mannequin has rightly flagged the picture on violence.  

Word: You’ll be able to ignore the classes and use the class scores to achieve management over the brink, this will make the moderation extra lenient or strict.  

Potential Use Instances

OpenAI omni moderation can very properly be used at locations requiring content material scrutiny.

• Chatbots: Filter dangerous inputs earlier than sending to LLM.  
• Picture Evaluation: Detect dangerous pictures beforehand.  
• Social Media: Flag hate speech and abusive content material.  
• Stay Streaming: Detect unsafe video frames utilizing moderation checks.  
• Multilingual Apps: Enhance moderation for different language inputs. 

Conclusion 

The omni-moderation-latest mannequin from OpenAI supplies an efficient security layer for LLM-based methods with help for each textual content and picture moderation. Whereas different OpenAI fashions can be utilized for moderation, this endpoint is particularly made for moderation and is totally free to make use of. Alternate options embody Azure AI Content material Security, which helps textual content and picture moderation with customizable security thresholds and enterprise integrations. 

Often Requested Questions

Captivated with expertise and innovation, a graduate of Vellore Institute of Expertise. At the moment working as a Knowledge Science Trainee, specializing in Knowledge Science. Deeply fascinated about Deep Studying and Generative AI, wanting to discover cutting-edge methods to unravel complicated issues and create impactful options.