Information missing in the documentation about ngwaf HOT 9 OPEN

Hi @rokasz thanks for reaching out, didn't realise the instructions weren't clear enough, will work on rewording em when I get the time to do so.
For v2.0 we've completely overhauled the ML pipeline and hosted it on AWS.

You are right about deploying sagemaker first (check out our terraform scripts to help you with that). I will check again to see if the base model is working well, all required information is from your setup.

For Telegram config, please do launch your own notification bot with BotFather and change the information required as per the new bot.

Let me know if you have any other questions 👯

---- Note to team: will open a jira ticket to maybe do up a readme for each folder.

from ngwaf.

@FA-PengFei Hello, big brother. I encountered some problems during the deployment process using the run.sh script. Is it convenient to leave a contact information? I want to ask you for advice

from ngwaf.

@xiaorui16888 could you open a separate issue for this?

from ngwaf.

@FA-PengFei , I was able to deploy all sagemaker stuff using terraform successfully. Then I am trying to launch the NGWAF application stack on Docker on Mac OS, but I am not able to make it work. Is the stack ready to be functional with the AWS Sagemaker setup? Here are the issues I am facing:

1. When opening http://localhost:8080 on my browser, I get "Internal Server Error" (500). I see the following errors in the log of the ngwaf-app_waf_1 container:
   src_ip: 172.21.0.1 [ * ] Printing Payload(s) Payload 1 = /favicon.ico Payload 2 = host Payload 3 = localhost:8080 Payload 4 = connection Payload 5 = keep-alive Payload 6 = sec-ch-ua Payload 7 = "Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114" Payload 8 = sec-ch-ua-mobile Payload 9 = ?0 Payload 10 = user-agent Payload 11 = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Payload 12 = sec-ch-ua-platform Payload 13 = "macOS" Payload 14 = accept Payload 15 = image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8 Payload 16 = sec-fetch-site Payload 17 = same-origin Payload 18 = sec-fetch-mode Payload 19 = no-cors Payload 20 = sec-fetch-dest Payload 21 = image Payload 22 = referer Payload 23 = http://localhost:8080/ Payload 24 = accept-encoding Payload 25 = gzip, deflate, br Payload 26 = accept-language Payload 27 = en-US,en;q=0.9,lt;q=0.8 Payload 28 = cookie Payload 29 = sid=Fe26.2**bac3c81283a02551f537e9c53cbcb3f10a434380052aeadf3a9b10815c4be3f6*nmQlyN3VZ5oHuLgaYZjhVQ*4LfIv-m7Bu3BBDuNiCwGzumf7wdrr8zIQQ9R5zCckCXIcadq_5gQ_4NIsht5n99W4OGRhodpzLsIwdqz8QPfd7E7rrtdD8GETbipgTXZvTWsRYhdqfc6q7HvrsLEzBwmZDehd_rqfBiqLvAZgNiKcdPizwE8gxsi0vCK117uF9xFNKKTxb9Zm_tkhwTjHB6wDL_cCwmf-siEBtR4Mh7gMDq_Q6E_z9HQ31qW4HMvhVw**3a8857928ca4f54e2e89d7ef7a21bed490039a28011999cab9ae4046354d6363*pojcOYejBQXXyBojAe5a2w9qlSZLRz8PloSQdLrr504 Payload 30 = sid Payload 31 = Fe26.2**bac3c81283a02551f537e9c53cbcb3f10a434380052aeadf3a9b10815c4be3f6*nmQlyN3VZ5oHuLgaYZjhVQ*4LfIv-m7Bu3BBDuNiCwGzumf7wdrr8zIQQ9R5zCckCXIcadq_5gQ_4NIsht5n99W4OGRhodpzLsIwdqz8QPfd7E7rrtdD8GETbipgTXZvTWsRYhdqfc6q7HvrsLEzBwmZDehd_rqfBiqLvAZgNiKcdPizwE8gxsi0vCK117uF9xFNKKTxb9Zm_tkhwTjHB6wDL_cCwmf-siEBtR4Mh7gMDq_Q6E_z9HQ31qW4HMvhVw**3a8857928ca4f54e2e89d7ef7a21bed490039a28011999cab9ae4046354d6363*pojcOYejBQXXyBojAe5a2w9qlSZLRz8PloSQdLrr504 {"message":"Forbidden"} INFO: 172.21.0.1:59580 - "GET /favicon.ico HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/wafuser/.local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi result = await app(self.scope, self.receive, self.send) File "/home/wafuser/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__ return await self.app(scope, receive, send) File "/home/wafuser/.local/lib/python3.8/site-packages/fastapi/applications.py", line 269, in __call__ await super().__call__(scope, receive, send) File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__ await self.middleware_stack(scope, receive, send) File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__ raise exc File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__ await self.app(scope, receive, _send) File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__ raise exc File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__ await self.app(scope, receive, sender) File "/home/wafuser/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__ raise e File "/home/wafuser/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__ await self.app(scope, receive, send) File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/routing.py", line 670, in __call__ await route.handle(scope, receive, send) File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/routing.py", line 266, in handle await self.app(scope, receive, send) File "/home/wafuser/.local/lib/python3.8/site-packages/starlette/routing.py", line 65, in app response = await func(request) File "waf.py", line 23, in proxy return await request_handler(starlette_request) File "waf.py", line 193, in request_handler is_blocked = await inspect_request(starlette_request) File "waf.py", line 112, in inspect_request score, payload = mlwaf.prediction(payload_check) File "/waf/WafApp/classifier.py", line 56, in prediction score = res.json()["score"] KeyError: 'score'

2. The ngwaf-app_custom-honey container fails to start and keeps in Exited state with the following errors in the log:
   Traceback (most recent call last): File "app.py", line 3, in <module> from fastapi import FastAPI, Request File "/home/wafuser/.local/lib/python3.8/site-packages/fastapi/__init__.py", line 7, in <module> from .applications import FastAPI as FastAPI File "/home/wafuser/.local/lib/python3.8/site-packages/fastapi/applications.py", line 16, in <module> from fastapi import routing File "/home/wafuser/.local/lib/python3.8/site-packages/fastapi/routing.py", line 22, in <module> from fastapi import params File "/home/wafuser/.local/lib/python3.8/site-packages/fastapi/params.py", line 6, in <module> from typing_extensions import Annotated, deprecated ImportError: cannot import name 'deprecated' from 'typing_extensions' (/home/wafuser/.local/lib/python3.8/site-packages/typing_extensions.py)

3. I was not sure, which S3 bucket to specify in the waf-admin-secrets.env file, as there are multiple in S3, so I set this:
   BUCKET_NAME="ngwaf-sagemaker-terraform"
   How is this bucket supposed to be used by the application? I saw in AWS, that public access is blocked, but I am running Docker on my laptop, so do I need to open the access to the bucket over the Internet?
   Also, I needed to add the Telegram settings into the waf-admin-secrets.env file as well, because the ngwaf-app_waf_admin_1 container keeps restarting due to Telagram variables missing.

Perhaps you have a newer code available, but did not update the github repo yet? Please advise.
Thank you and really appreciate your efforts with this product.

from ngwaf.

Hi @rokasz ,
Have you added the AWS credentials generated from the terraform output file into the environment variables file for the docker files?

from ngwaf.

@duemaster , yes, I did. Here are my config files:

waf-secrets.env:
# ML MODEL API ENDPOINT
API_ENDPOINT="https://xyxyxyxyx.execute-api.ap-southeast-1.amazonaws.com/terraform_deploy"
API_KEY="rboTda8ViXXXXXXXXXXXXXXXXXXXX6QqAFi%"
# TELEGRAM CONFIG
TELEGRAM_CHAT_ID="63XXXXXX93"
TELEGRAM_TOKEN="AAFecw5hXXXXXXXXXLsbPXMKOeOrmdQcFw4"

waf-admin-secrets.env:
# ML MODEL API ENDPOINT
API_ENDPOINT="https://xyxyxyxyx.execute-api.ap-southeast-1.amazonaws.com/terraform_deploy"
API_KEY="rboTda8ViXXXXXXXXXXXXXXXXXXXX6QqAFi%"
BUCKET_NAME="ngwaf-sagemaker-terraform"

ACCESS_KEY_ID="AKIXXXXXXXXXXXP4BFFJ"
SECRET_ACCESS_KEY="3pXXXXXXXXxXXXXJi/J9Q4nwPnmAXSSeiO9"

SQL_USERNAME="ngwafuser"
SQL_PASSWORD="ngwafpassword"
SQL_SERVER="mysql"
SQL_DATABASE_NAME="ngwaf"

# TELEGRAM CONFIG
TELEGRAM_CHAT_ID="63XXXXXX93"
TELEGRAM_TOKEN="AAFecw5hXXXXXXXXXLsbPXMKOeOrmdQcFw4"

from ngwaf.

@FA-PengFei , @duemaster , do you have any update about the above? Are you still maintaining the project? Was anyone besides you successful in deploying and running it?
Thanks a lot.

from ngwaf.

@rokasz yeap sorry we took so long, we got real busy supporting a new project that we pushed out couple of months back, we are still supporting NGWAF for bugs and issues just not with further updates to the open source code base. I went through your logs, it seems to purely be dependencies issues on your docker containers. I will run a test this upcoming week to see if I can replicate those and get back to you with a fix if it is fixable on our side. The S3 bucket is where the code will retrieve your latest retrained model so the admin container do need to reach that. @duemaster let me know if I'm missing anything for this.

from ngwaf.

Hi @rokasz , could if you looked at the error message for pt 1, it seems that the API gateway is returning forbidden. Could you verify by checking if the correct API key is provided and also add some debug messages here:

You could try printing the API endpoint and verify that the endpoint works.

from ngwaf.