Code Monkey home page Code Monkey logo

Comments (7)

jiangwenyuan avatar jiangwenyuan commented on May 9, 2024

Hi, thanks for the reporting:)

I've did a little investigation and confirmed bug 1 but cannot confirm bug2/3.

bug 2:

if you didn't set key first, 404 is returned:

$ wrk -t2 -c200 -d30s http://127.0.0.1/nosql/key11
Running 30s test @ http://127.0.0.1/nosql/key11
  2 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.05ms    1.16ms 203.06ms   92.31%
    Req/Sec    24.27k     1.20k   28.12k    79.83%
  1450289 requests in 30.03s, 62.24MB read
  Non-2xx or 3xx responses: 1450289
Requests/sec:  48286.80
Transfer/sec:      2.07MB

if you have set key already:

$ curl -d vv http://127.0.0.1/nosql/key11
$ wrk -t2 -c200 -d30s http://127.0.0.1/nosql/key11
Running 30s test @ http://127.0.0.1/nosql/key11
  2 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.02ms    2.41ms 205.12ms   84.18%
    Req/Sec    13.87k     2.88k   17.05k    80.67%
  828870 requests in 30.05s, 70.35MB read
Requests/sec:  27582.70
Transfer/sec:      2.34MB

bug 3:

I use this script:

local url = '/nosql'
local methods = {
    'GET',
    'POST',
    --'DELETE',
}
math.randomseed(os.time())
request = function()
    local r = math.random(1, 650000)
    wrk.body = r .. "\n"
    return wrk.format(
    methods[r % 2],
    url
    )
end

cannot reproduce it.

could you please help me reproduce it?

from nuster.

Dexus77 avatar Dexus77 commented on May 9, 2024

Bug 2-3 is really tricky only occurs in a certain configuration. Today, I sketched out the code to check the bug and it did not work for me, it was ok.

If you checked with the configuration of nuster for a bug 1 where on one backend the cache is disabled, and chache on the second the bugs 2-3 are disabled in this configuration do not show themselves.

Try to enable caching on both backends, but even in this configuration I can not yet achieve such a 100% effect as yesterday. The bug 2 is very rarely seen by Nuster, it breaks the connection and issues Internal Server Error. To achieve leakage of the processor while it was not possible.

Very sample code is just for you to understand.

#!/usr/local/python/3.7.0/bin/python3
import asyncio
import uvloop
import aiohttp
from aiohttp import web

url = 'http://127.0.0.1/nosql/key1'
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())


async def func(request):
    async with aiohttp.ClientSession() as session:
        try:
            async with session.post(url, data='value1', ) as response:
                if response.status != 200:
                    print('status: ', response.status)
        except Exception as ex:
            print('Set exception type: ', type(ex).__name__)

        try:
            async with session.get(url) as response:
                if response.status != 200:
                    print('status: ', response.status)
        except Exception as ex:
            print('Get exception type: ', type(ex).__name__)
    return web.Response(text="ok")


async def init():
    app = web.Application()
    app.router.add_route('GET', '/', func)
    return app

web.run_app(init(), port=8111)

The bugs surfaced under the coded code, which I already loaded in the wrk.
I tested on 2 frameworks, in both there were connection breaks and processor utilization.

If I get to find the configuration in which 100% worked the bug I will write.

from nuster.

Dexus77 avatar Dexus77 commented on May 9, 2024

I was able to find out why there is a processor leak.
It is necessary as a value to put a dict and load wrk.

#!/usr/local/python/3.7.0/bin/python3
import asyncio
import uvloop
import aiohttp
from aiohttp import web

url = 'http://127.0.0.1/nosql/key1'
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

async def func(request):
    async with aiohttp.ClientSession() as session:
        data = {'app': 11, 'value': 'sdf sf asf fhw adsf rhrtw'}
        try:
            async with session.post(url, data=data, ) as response:
                if response.status != 200:
                    print('status: ', response.status)
        except Exception as ex:
            print('Set exception type: ', type(ex).__name__)

        try:
            async with session.get(url) as response:
                if response.status != 200:
                    print('status: ', response.status)
        except Exception as ex:
            print('Get exception type: ', type(ex).__name__)
    return web.Response(text="ok")

async def init():
    app = web.Application()
    app.router.add_route('GET', '/', func)
    return app

web.run_app(init(), port=8111)

from nuster.

jiangwenyuan avatar jiangwenyuan commented on May 9, 2024

Thanks, but I still cannot reproduce bug2/3, your app only gives me this error which i think has nothing to do with nuster, I tried with nginx(only get part) , and can get this error too.

Get exception type:  CancelledError
Unhandled exception
Traceback (most recent call last):
  File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 410, in start
    await resp.prepare(request)
  File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_response.py", line 300, in prepare
    return await self._start(request)
  File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_response.py", line 608, in _start
    return await super()._start(request)
  File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_response.py", line 367, in _start
    await writer.write_headers(status_line, headers)
  File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/http_writer.py", line 110, in write_headers
    self._write(buf)
  File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/http_writer.py", line 67, in _write
    raise ConnectionResetError('Cannot write to closing transport')
ConnectionResetError: Cannot write to closing transport

BTW, bug 1 has beed fixed in 2d94461

from nuster.

Dexus77 avatar Dexus77 commented on May 9, 2024

Yes this is what I was talking about. The bug 2 nuster terminates the connection and the aiohttp client can not write to it. CancelledError can still be a ServerDisconnect. To exclude errors aiohttp checked on another web framework the situation was similar. Did you notice after these errors that Nuster did not consume 100% of the CPU? (Bag3)

from nuster.

jiangwenyuan avatar jiangwenyuan commented on May 9, 2024

Hi, if you try this code:

import asyncio
import uvloop
import aiohttp
from aiohttp import web

url = 'http://127.0.0.1/nosql/key1'
url = 'http://127.0.0.1/index.html'
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

async def func(request):
    async with aiohttp.ClientSession() as session:
        try:
            async with session.get(url) as response:
                if response.status != 200:
                    print('status: ', response.status)
        except Exception as ex:
            print('Get exception type: ', type(ex).__name__)
    return web.Response(text="ok")

async def init():
    app = web.Application()
    app.router.add_route('GET', '/', func)
    return app

web.run_app(init(), port=8111)

and run nginx to serve 'http://127.0.0.1/index.html' , the error still appears.

Did you notice after these errors that Nuster did not consume 100% of the CPU? (Bag3)

Do you mean consume 100% cpu instead of did not consume ?

No, nuster did not consume 100% of the CPU.

from nuster.

jiangwenyuan avatar jiangwenyuan commented on May 9, 2024

Hi, I've being trying to reproduce bug2/3, but still cannot reproduce. I'm closing it now, feel free to reopen it.

BTW bug1 has fixed and merged into master.

thanks

from nuster.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.