Code Monkey home page Code Monkey logo

aisg-online-safety-challenge-submission-guide's Introduction

AI Singapore Online Safety Prize Challenge Submission Guide

Participants must submit a compressed Docker container in the tar.gz format via the challenge platform. This repository serves as a step-by-step guide to help participants create a valid submission for the Online Safety Prize Challenge.

While the proper term for the Docker generated artefacts is "Docker images", we will use the term "Docker container" instead to disambiguate it from the [computer] images that serve as input to the challenge.

Getting Started

We are using Docker for this challenge so that participants can choose their preferred programming languages and open source dependencies to create the best performing detection models.

To build and run GPU accelerated Docker containers, please install the NVIDIA Container Toolkit in your development environment.

Technical Details

Hardware/Software Specifications

All participants' compressed Docker containers will be executed on virtual machines with the following resource allocation:

vCPU Mem (GB) GPU tmpfs (GiB)
6 60 V100 16GB VRAM 5

This will be reflected in the docker run command options. Participants may specify different settings for their own testing purposes, but these will not be reflected in the official run-time environment for scoring.

The general software specification

  • Instruction Set: x86-64
  • Ubuntu 22.04
  • NVIDIA Driver Version: 535
  • Docker Version: 24.0.7
  • NVIDIA Container Toolkit: 1.14.4-1

IMPORTANT NOTE: The following instructions relating to Docker assumes our general software specification.

Submission Specification Guidelines

This section will cover the following important guidelines on building your solution for submission:

  1. A brief overview of the dataset;
  2. The required input format for your submitted Docker container and the output format from it;
  3. The maximum resources of a Docker container for each submission;
  4. The performance metrics for this challenge;
  5. Instructions on how to run this repository and create your own submission.

Dataset, Input, Output

Dataset:

The dataset comprises about 1700 images in PNG format with file extension .png. The directory containing the dataset will be mounted to your Docker container as read-only at the mount point of /images.

Input to Docker Container:

Your solution must use stdin, where each line from stdin corresponds to a file path to a single image. This simulates a real-world situation where your solution is called on a per-image basis. An example of a line of input from stdin might look like /images/9ad157be-f32f-4770-b409-8c4650478f5b.png .

Further details on how this is done for a Python-based Docker solution can be found in Usage of sample submission and Creating your own submission.

Output (stdout, stderr) to Container:
Solution output: stdout

Your solution must use stdout to output the result of your analysis for each line of input from stdin.

The output format per line of input of stdin must include:

  • The probability that the meme is harmful with 4 decimal places of precision. We will refer to this as proba.
  • An integer, where 1 is output if the meme is deemed harmful, and 0 if it is benign. We will refer to this as label.

Both predictions must be separated by a single tab character (\t), and terminated with a single new line character (\n).

Examples of implementation for the formatting of one line of output in Python can be seen below.

# Example 1:
output = f"{proba:.4f}\t{label}\n"

# Example 2 (more C-like)
output = "%.4f\t%d\n" % (proba, label)

sys.stdout.write(output)

Below shows an example of how stdout will look like after having processed (in this case) 5 lines of input from stdin:

0.8232	1
0.7665	1
0.3241	0
0.1015	0
0.9511	1

Further details on how this is done for a Python-based Docker solution can be found in Usage of sample submission and Creating your own submission.

Remember that there has to be one output format conforming line in stdout for every input line from stdin. Please do not attempt to skip any.

Failure to do so may result in inaccurate scoring of your results.

Logging output: stderr

Your solution must use stderr for the writing of any logs to assist you in determining any programming errors within your solution. Logs have an implied file size limit to prevent abuse. Failure to keep within this limit through excessive logging will result in an error in your solution.

Further details on how this is done for a Python-based Docker solution can be found in Usage of sample submission and Creating your own submission.

Non-compliance may result in premature termination of your solution with a Resource Limit Exceeded error.

Logs may be obtained only on a case-by-case basis. Requests can be made over at the discussion board, but the fulfilment of the request shall be at the discretion of the organisers.

Performance Metric Details

The performance metrics used are Area Under the Curve of the Receiver Operating Characteristic (AUROC) and accuracy. Both metrics will be displayed on the leaderboard.

  1. The proba output is used to calculate AUROC and used to determine your ranking on the leaderboard.
  2. The label output is used to calculate accuracy, and used as secondary metric in the event of a tie.

Docker Container Details

Max Docker Container Size

Your solution upon saving using docker save must not exceed the maximum file size of 25 GiB.

Max Docker Container Runtime

Your solution must not exceed 2.5 hours of runtime to process around 1700 images.

Submitted Docker Container Isolation

All submitted Docker containers are executed in a network isolated environment where there is no internet connectivity, nor access to any other external resources or data beyond what the container has.

As such, your solution must have all necessary modules, model weights, and other non-proprietary dependencies pre-packaged in your Docker container.

Non-compliance will result in your Docker container facing issues/error when in operation.

Example: Usage of sample submission

Pre-condition: Create the isolated Docker network

Before trying out the sample submission or creating your own submission, you will need to create a local Docker network to simulate the environment setup for the execution of solutions.

Run the following command to create your own isolated Docker network. If it is already created, as indicated by the output of docker network ls, you can skip this step.

ISOLATED_DOCKER_NETWORK_NAME=exec_env_jail_network
ISOLATED_DOCKER_NETWORK_DRIVER=bridge
ISOLATED_DOCKER_NETWORK_INTERNAL=true
ISOLATED_DOCKER_NETWORK_SUBNET=172.20.0.0/16

docker network create \
    --driver "$ISOLATED_DOCKER_NETWORK_DRIVER" \
    $( [ "$ISOLATED_DOCKER_NETWORK_INTERNAL" = "true" ] && echo "--internal" ) \
    --subnet "$ISOLATED_DOCKER_NETWORK_SUBNET" \
    "$ISOLATED_DOCKER_NETWORK_NAME"

Clone this repository and navigate to it

git clone https://github.com/AISG-Technology-Team/AISG-Online-Safety-Challenge-Submission-Guide.git

Change into the sample submission (sample_submission) directory

cd sample_submission

Build the sample Docker container

You can add a --no-cache option in the docker build command to force a clean rebuild.

docker build -t sample_container .

Please take note that the "." indicates the current working directory and should be added into the docker build command to provide the correct build context.

Test sample Docker container locally

Please ensure you are in the parent directory of sample_submission before executing the following command. The $(pwd) command in the --mount option yields the current working directory. The test is successful if no error messages are seen and a stdout.csv is created in the local_test/test_output directory.

Alter the options for --cpus, --gpus, --memory to suit the system you are using to test.

ISOLATED_DOCKER_NETWORK_NAME=exec_env_jail_network

cat local_test/test_stdin/stdin.csv | \
docker run --init \
        --attach "stdin" \
        --attach "stdout" \
        --attach "stderr" \
        --cpus 2 \
        --gpus "device=0" \
        --memory 4g \
        --memory-swap 0 \
        --ulimit nproc=1024 \
        --ulimit nofile=1024 \
        --network exec_env_jail_network \
        --read-only \
        --mount type=bind,source="$(pwd)"/local_test/test_images,target=/images,readonly \
        --mount type=tmpfs,destination=/tmp,tmpfs-size=5368709120,tmpfs-mode=1777 \
        --interactive \
        sample_container \
 1>local_test/test_output/stdout.csv \
 2>local_test/test_output/stderr.csv

Please note that the above docker run command would be equivalent to running the following command locally:

cat local_test/test_stdin/stdin.csv | \
    python3 sample_submission/main.py \
        1>local_test/test_output/stdout.csv \
        2>local_test/test_output/stderr.csv

(Optional) Adding own images

You can add a few more custom images (of the PNG format with file extension .png) into local_test/test_images. After doing that, you can regenerate the stdin.csv located in local_test/test_stdin using the following command.

cd utils

# Using the default location for the test_images and output_folder for storing stdin.csv
python3 gen_input.py --img_folder ../local_test/test_images --output_folder ../local_test/test_stdin

You can then re-run the Test sample Docker container locally

Compress your sample container to .tar.gz format using docker save

docker save sample_container:latest | gzip > sample_container.tar.gz

Upload container

The final step would be to submit the compressed Docker container file (sample_container.tar.gz in this example) on to the challenge platform, but since this is only the sample with no actual logic, we will not do so.

Please note that if you do submit this sample, it will still take up one count of your submission quota.

Example: Creating your own submission

The process of creating your own submission would be very similar to using the aforementioned sample submission.

Create a project directory and navigate into it

mkdir Online-Safety-Challenge && cd Online-Safety-Challenge

Create a main file

The main file has to be able to interact with standard streams such as stdin, stdout, and stderr.

In general, the main file should have the following characteristics:

  1. Read the PNG images from the file paths obtained from stdin (one file path per input line);
  2. Predict the probability that the image is a harmful meme, up to 4 decimal places (proba as referred to in the Submission Specification Guidelines);
  3. Decide if the image is a harmful meme (1), or a benign meme (0) (label as referred to in the Submission Specification Guidelines);
  4. Output a single line with to stdout for each line of stdin conforming to the Submission Specification Guidelines;
  5. Use stderr to log any necessary exceptions/errors.

Note:

Please ensure that the prediction output to stdout is in the same order as the stdin because the order in which stdout is assessed strictly follows the order of the input from stdin.

You must use /tmp within your Docker container for any temporary files for processing. This is because the Docker container will be executed with the options:

  • --read-only which sets the root file-system as read only.
  • --tmpfs /tmp which sets a fixed /tmp directory for any app to write to.

You may refer to the main.py of the sample submission as an example of a main file.

Create a Dockerfile

You may use the sample Dockerfile provided for you. However, please install the relevant dependencies required for your detection model. Additionally, you may wish to change the ENTRYPOINT if you are using another main file or if you prefer to use a shell script:

ENTRYPOINT ["bash","/path/to/your/main.sh"]

If you are not familiar with how to build a Dockerfile, please refer to the official documentation for more information.

Build your Docker container using docker build

docker build -t your_container .

Please take note that the "." indicates the current working directory and should be added into the docker build command to provide the correct build context.

Test your Docker container locally

1. Create a test directory outside of your project directory

mkdir local_test && cd local_test

2. Create input and output directory within your test directory

mkdir test_images test_output

3. Add test images (PNG format with file extension .png) into test_images directory

4. Generate the input file (.csv).

You can use the following script utils/gen_input.py in this guide to generate the stdin.csv which will be used as the source for stdin.

# Using the default location for the test_images and output_folder for storing stdin.csv
python3 gen_input.py --image_folder ../local_test/test_images --output_folder ../local_test/test_stdin

The script will create a stdin.csv file within local_test/test_stdin which will be used as the source of input for stdin for your Docker container.

5. Test your container using docker run

Please ensure you are in the parent directory before executing the following command. The $(pwd) command in the --mount option yields the current working directory. The test is successful if no error messages are seen and a stdout.csv is created in the local_test/test_output directory.

Alter the options for --cpus, --gpus, --memory to suit the system you are using to test.

ISOLATED_DOCKER_NETWORK_NAME=exec_env_jail_network

cat local_test/test_stdin/stdin.csv | \
docker run --init \
        --attach "stdin" \
        --attach "stdout" \
        --attach "stderr" \
        --cpus 2 \
        --gpus "device=0" \
        --memory 4g \
        --memory-swap 0 \
        --ulimit nproc=1024 \
        --ulimit nofile=1024 \
        --network exec_env_jail_network \
        --read-only \
        --mount type=bind,source="$(pwd)"/local_test/test_images,target=/images,readonly \
        --mount type=tmpfs,destination=/tmp,tmpfs-size=5368709120,tmpfs-mode=1777 \
        --interactive \
        your_container \
 1>local_test/test_output/stdout.csv \
 2>local_test/test_output/stderr.csv

Compress your Docker container to .tar.gz format using docker save

docker save your_container:latest | gzip > your_container.tar.gz

Upload your container

Submit your your_container.tar.gz file onto the challenge platform. Please note that when you do this, it will take up one count of your submission quota.

aisg-online-safety-challenge-submission-guide's People

Contributors

mtmak-aisg avatar nameless1117 avatar ywkuan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

aisg-online-safety-challenge-submission-guide's Issues

Baseline - petergro - Change tmpfs tmp mount?

Hello dear challenge team,

I wanted to test some local solution on the server but I've run in a bit of an issue. Docker has had a bug with mounting tmpfs volumes, making them non-executable (despite being marked as such) for the past 7 years or longer, see moby/moby#35890 or moby/moby#35890 .
For the solution however I'd like to run libraries from the /tmp folder. Therefore I would like to ask if containers could be executed with --tmpfs /tmp:exec instead of --mount type=tmpfs,destination=/tmp,tmpfs-size=5368709120,tmpfs-mode=1777 or some folder be mounted with something else other than tmpfs, or any other solution that allows files to be executed from a folder.

Thank you and best

JK-Piece -- Request for logs

Can you please provide logs for the following two submissions

1-) 7b4f3fe5-b6e0-4ad6-b456-07714b7b2ab0

2-) c7640ddd-a8a8-4124-8aa6-dac63161656d

Thanks.

Chaos - xyz - Request for stderr log

Hello,

We recently made a submission with the ID 5212ad53-3943-4732-804b-0c4cc6cf4c70 and encountered an 'Output Format Error'. This was unexpected as our local tests seemed to conform to the output format requirements.

To better understand the issue, could we kindly request the stderr log for our submission?

Thank you for your time and assistance.

something - Aradhye2002 - Output format error

I am getting an output format error on submitting. However I have used the main.py in the sample_submission only, making changes to the classifier(., .) function. I have also verified that the output is correct by running the container locally.

Could you please send over the logs and convey what exactly is the reason for an output error?

Thanks

R2Studio - huyenbk_vinuni - Submission failed

Hi, I've encountered an issue where my last 4 upload attempts on the same device have been unsuccessful, which is puzzling since I've had successful uploads previously. Additionally, these unsuccessful tries are being deducted from my total submission quota, which is a waste of resource. Could these failed attempts possibly be removed from my submission count?

JK-Piece--Request to show the number of images processed before runtime limit exceeded error.

Dear technical support,

My previous submissions exceeded the runtime limit, and therefore an error was raised.
I can only see Time Limit Exceeded.
Would it be possible to show how many images are left before the time limit is exceeded? This would help design an optimal model in terms of runtime.

Instead of showing "Running", it would be helpful to have a progress bar like Processed 32/1700

Submission Upload Failed

Hello, I want to ask if "Submission Upload Failed" takes up team's submission quota? I hope not, since the network fluctuation is easy to occur when uploading the submission.

AI Singapore Online Safety Prize Challenge - Discussion Board

AI Singapore Online Safety Prize Challenge - Discussion Board

Welcome to the participant discussion board for the "Online Safety Prize Challenge" (OSPC), a 10-week competition designed to inspire participants to develop a sophisticated multi-modal classification model capable of discerning between benign and harmful memes. Given the sensitive nature of the content, participants are tasked with independently sourcing relevant datasets, devising data augmentation methods, or utilizing automated meme generation techniques to curate an appropriate dataset for model training and testing.

Important Information for Participants:

This discussion board is exclusively reserved for OSPC participants. Discussion threads started by non-participants will be promptly deleted. The OSPC team is committed to responding to queries as soon as practically possible.

Guidelines for Submitting Questions:

When submitting a question, adhere to the specified naming convention outlined below to facilitate efficient user identification. Ensure that the details provided align with the registration information on the Challenge Landing Page. Please note that questions not conforming to the naming conventions will not be addressed and will be promptly deleted.

Issue Naming Convention:

[Team Name] - [Participant User ID] - [Question Topic]

Example:

For an individual from the team 'RedBadgers', with a user ID 'Happy_badger88', seeking information about available datasets, the issue should be formatted as follows:

Issue Name: RedBadgers - Happy_badger88 - Where can I find available Datasets?

We appreciate your adherence to these guidelines, and we look forward to a robust and engaging discussion throughout the OSPC. For additional details and updates, please refer to the Challenge Landing Page.

AIris - CjangCjengh - Question about the combination of scores

image Hello, I noticed a graph on the competition's official website regarding the combination of scores. It states that AUROC accounts for 70% of the final score. If it does not violate any confidentiality principles, could you please explain the specific calculation method? For example, if a team has an AUROC of 0.6, does that mean its contribution to the final score is 0.6 * 0.7 = 0.42? Or are there more complex calculation mechanisms involving weighted algorithms and ranking scores?

Offensive memes - aliencaocao - On handling of TLE

Hi, we would like to know if a docker does not quit itself for some reason but all the stdout are already complete for each line of stdin and in the right format, does this become a time limit exceeded or output format error?

Is there anyway AISG could take whatever outputted at end of 2.5hrs runtime and check if it is the right format?

offensive memes - aliencaocao - batched inference?

I am writing to check if batched inference is allowed for the docker image, or must it strictly be single-sample per batch? Asking because speed difference is very large and we could read all stdin at once then output all stdout at once too. The competition github repository did not specify on this and such is possible based on the example scripts given.

UQ-GD Team Request for Logs

Hi, Could we please have the logs for submission e4f0c282-6f41-425d-b461-a925ffb5401b ?
We are probably having problems with running code on the GPU because everything works when we run the container locally (we are using the CUDA base image from nvidia). Are there any known incompatibilities with keras_ocr?

Could we also get feedback about the output for this same submission? It is formatted correctly when we run the container locally so we were wondering if the container is just crashing or if the output is incorrectly formatted.

Thanks

Baseline - petergro - request for logs

Hello,

I'd like to request stderr logs for 6fe34f8a-599a-4a58-90b8-ac54db3f9c34, 04472c06-1b6e-4b2d-8829-462c2aab1dca and a72e1b8b-c2e0-41d3-b76c-10df0cdcdea8 .
If they are empty, no need to send them.

Thank you

offensive memes - aliencaocao - request for logs

Hi,
We would like to request for logs for our recent run 8d46664b-66b8-47f0-87b1-d0e8472bc15c

It has been tested using the same docker command on another V100 16GB machine and no issues were met. We dont know why is AISG's VM different from ours. We have also made a successful submission before and used the same dockerfile base but just swapped the model weights.

Here is a picture of it working on our side with same GPU
image

GPU Compatibility on Submission Platform - UQ-GD

Hi, I'm from the UQ-GD team. We are encountering GPU compatibility issues on the challenge platform, affecting our progress. Our code runs on a CPU locally, but fails on the platform's GPU environment, likely due to compatibility problems. We're using the Nvidia Docker image with CUDA and cuDNN as specified, yet errors persist.

Could you provide guidance or a GPU-specific code sample? This would allow us to focus on the challenge tasks rather than setup issues.

Request for more details on errors - JKPiece

My recent submission ran for approximately 2 hours, then displayed "Error". Would it be possible to provide more details on the error so I can make a valid submission?

Fingerprint: 8d6d871d-cbf4-456c-a52d-89af458ba549

offensive memes - aliencaocao - Evaluation timing

Hi,

We noticed that the submission shows as running even when the backend is extracting the docker image. Does that count into our 2.5hr of runtime, or is that 2.5hr starting from the launch of docker container?

JK-Piece - JK-Piece - Request for logs

Unfortunately, I still cannot get my submission scoring. Here is the new fingerprint (recent submission) for which I would like to have the logs: 535ae95e-7513-458c-a72d-016cce3fd8bf

Originally posted by @Jean-KOUAGOU in #9
Please submit a separate issue for new queries in future. Thank you.

Hi Jean-KOUAGOU, logs have been sent to your email.

JK-Piece-JK-Piece-The logs do not show any error in my code

Hi, thanks for providing the logs for my submission 535ae95e-7513-458c-a72d-016cce3fd8bf.
However, there is no error. I can only see a warning which comes from the images themselves. Below are some of the warnings I can see:

First, there was this warning in my previous submission bdd790f6-47b1-4fe6-abdb-32ecc30fe5bb:
"libpng warning: iCCP: known incorrect sRGB profile".

Then, I fixed it by installing imagemagic: RUN apt install imagemagick-6.q16 was added to my Docker file.
Afterwards, I used os.system(f"convert {image_path} /tmp/img.png") to convert input images. Next, I use the new file "/tmp/img.png" instead of the original one.

Now in my recent submission for which you sent the logs, the following warning occurs:

"convert-im6.q16: CorruptImageProfile `/images/d0fdb61ae1bb65fa49a5747b7faa3fbf.png' (XMP) @ warning/profile.c/ValidateXMPProfile/1675."], ["convert-im6.q16: CorruptImageProfile `/images/293bb2aec900fb3f9753248e00f4a8b5.png' (XMP) @ warning/profile.c/ValidateXMPProfile/1675."], ["convert-im6.q16: CorruptImageProfile `/images/04583003b0a23c09c0f580247c16b6e8.png' (XMP) @ warning/profile.c/ValidateXMPProfile/1675."

So my point is that there is no error, but only warnings. Can you please assist with this?

Is it the case that the platform considers warnings as errors?

Output format error UQ-GD team

Hi, our team is getting an output format error.

Could you please send over the logs and stdout and convey what exactly is the reason for an output error?
After building the docker locally on our machine everything works fine, that is why we are contacting you.
Our submission id is the following: a8fa6fb9-e3eb-42d3-980c-82d2d25e1e5c

Thanks

Detective Orange - Weibin - Output format error

Hi, I followed the output rule and verified the format of stdout.csv is correct by running the container locally. So I don't know the reason for this error. Could you please send over the logs and convey what exactly is the reason for this output error? Thanks for your help!

offensive memes - aliencaocao - docker run specs

Hi,
We note that in readme you use

ISOLATED_DOCKER_NETWORK_NAME=exec_env_jail_network

cat local_test/test_stdin/stdin.csv | \
docker run --init \
        --attach "stdin" \
        --attach "stdout" \
        --attach "stderr" \
        --cpus 2 \
        --gpus "device=0" \
        --memory 4g \
        --memory-swap 0 \
        --ulimit nproc=1024 \
        --ulimit nofile=1024 \
        --network exec_env_jail_network \
        --read-only \
        --mount type=bind,source="$(pwd)"/local_test/test_images,target=/images,readonly \
        --mount type=tmpfs,destination=/tmp,tmpfs-size=5368709120,tmpfs-mode=1777 \
        --interactive \
        sample_container \
 1>local_test/test_output/stdout.csv \
 2>local_test/test_output/stderr.csv

Which limits the docker to only 2 cpus and 4gb RAM, while the technical spec listed is 6cpu and 60GB RAM. Which is accurate at evaluation?

Detective Orange - Weibin - Submission failed

Hi, I tried to upload my submission three times but all of them are failed. I'm confused about the reason. My previous submissions are successful. And all these submissions are uploaded on the same device in my lab, the network should be stable. Does anyone have the same situation as me? Thanks for your help.

offensive memes - sheepymeh - Request for logs

I'd like to request logs from our run a35d3475-37b0-4029-9002-b8bcbd7771ea. We believe that it was a CUDA OOM. We'd like to confirm whether this was the case, and whether the container was able to generate any outputs. Thank you.

Baseline - petergro - Request for clarification

Hello,

I received a 'load docker image error' on fbccd43f-887b-4bed-9a4c-bbf1896c9364 I would like to request clarification on why that is/what the errors were. From my tests it should all be within the challenge limitations.

Thank you

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.