Comments (7)
Hi, thanks for the reporting:)
I've did a little investigation and confirmed bug 1 but cannot confirm bug2/3.
bug 2:
if you didn't set key first, 404 is returned:
$ wrk -t2 -c200 -d30s http://127.0.0.1/nosql/key11
Running 30s test @ http://127.0.0.1/nosql/key11
2 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.05ms 1.16ms 203.06ms 92.31%
Req/Sec 24.27k 1.20k 28.12k 79.83%
1450289 requests in 30.03s, 62.24MB read
Non-2xx or 3xx responses: 1450289
Requests/sec: 48286.80
Transfer/sec: 2.07MB
if you have set key already:
$ curl -d vv http://127.0.0.1/nosql/key11
$ wrk -t2 -c200 -d30s http://127.0.0.1/nosql/key11
Running 30s test @ http://127.0.0.1/nosql/key11
2 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.02ms 2.41ms 205.12ms 84.18%
Req/Sec 13.87k 2.88k 17.05k 80.67%
828870 requests in 30.05s, 70.35MB read
Requests/sec: 27582.70
Transfer/sec: 2.34MB
bug 3:
I use this script:
local url = '/nosql'
local methods = {
'GET',
'POST',
--'DELETE',
}
math.randomseed(os.time())
request = function()
local r = math.random(1, 650000)
wrk.body = r .. "\n"
return wrk.format(
methods[r % 2],
url
)
end
cannot reproduce it.
could you please help me reproduce it?
from nuster.
Bug 2-3 is really tricky only occurs in a certain configuration. Today, I sketched out the code to check the bug and it did not work for me, it was ok.
If you checked with the configuration of nuster for a bug 1 where on one backend the cache is disabled, and chache on the second the bugs 2-3 are disabled in this configuration do not show themselves.
Try to enable caching on both backends, but even in this configuration I can not yet achieve such a 100% effect as yesterday. The bug 2 is very rarely seen by Nuster, it breaks the connection and issues Internal Server Error. To achieve leakage of the processor while it was not possible.
Very sample code is just for you to understand.
#!/usr/local/python/3.7.0/bin/python3
import asyncio
import uvloop
import aiohttp
from aiohttp import web
url = 'http://127.0.0.1/nosql/key1'
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def func(request):
async with aiohttp.ClientSession() as session:
try:
async with session.post(url, data='value1', ) as response:
if response.status != 200:
print('status: ', response.status)
except Exception as ex:
print('Set exception type: ', type(ex).__name__)
try:
async with session.get(url) as response:
if response.status != 200:
print('status: ', response.status)
except Exception as ex:
print('Get exception type: ', type(ex).__name__)
return web.Response(text="ok")
async def init():
app = web.Application()
app.router.add_route('GET', '/', func)
return app
web.run_app(init(), port=8111)
The bugs surfaced under the coded code, which I already loaded in the wrk.
I tested on 2 frameworks, in both there were connection breaks and processor utilization.
If I get to find the configuration in which 100% worked the bug I will write.
from nuster.
I was able to find out why there is a processor leak.
It is necessary as a value to put a dict and load wrk.
#!/usr/local/python/3.7.0/bin/python3
import asyncio
import uvloop
import aiohttp
from aiohttp import web
url = 'http://127.0.0.1/nosql/key1'
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def func(request):
async with aiohttp.ClientSession() as session:
data = {'app': 11, 'value': 'sdf sf asf fhw adsf rhrtw'}
try:
async with session.post(url, data=data, ) as response:
if response.status != 200:
print('status: ', response.status)
except Exception as ex:
print('Set exception type: ', type(ex).__name__)
try:
async with session.get(url) as response:
if response.status != 200:
print('status: ', response.status)
except Exception as ex:
print('Get exception type: ', type(ex).__name__)
return web.Response(text="ok")
async def init():
app = web.Application()
app.router.add_route('GET', '/', func)
return app
web.run_app(init(), port=8111)
from nuster.
Thanks, but I still cannot reproduce bug2/3, your app only gives me this error which i think has nothing to do with nuster, I tried with nginx(only get part) , and can get this error too.
Get exception type: CancelledError
Unhandled exception
Traceback (most recent call last):
File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 410, in start
await resp.prepare(request)
File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_response.py", line 300, in prepare
return await self._start(request)
File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_response.py", line 608, in _start
return await super()._start(request)
File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/web_response.py", line 367, in _start
await writer.write_headers(status_line, headers)
File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/http_writer.py", line 110, in write_headers
self._write(buf)
File "/home/jwy/.pyenv/versions/3.6.6/lib/python3.6/site-packages/aiohttp/http_writer.py", line 67, in _write
raise ConnectionResetError('Cannot write to closing transport')
ConnectionResetError: Cannot write to closing transport
BTW, bug 1 has beed fixed in 2d94461
from nuster.
Yes this is what I was talking about. The bug 2 nuster terminates the connection and the aiohttp client can not write to it. CancelledError can still be a ServerDisconnect. To exclude errors aiohttp checked on another web framework the situation was similar. Did you notice after these errors that Nuster did not consume 100% of the CPU? (Bag3)
from nuster.
Hi, if you try this code:
import asyncio
import uvloop
import aiohttp
from aiohttp import web
url = 'http://127.0.0.1/nosql/key1'
url = 'http://127.0.0.1/index.html'
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def func(request):
async with aiohttp.ClientSession() as session:
try:
async with session.get(url) as response:
if response.status != 200:
print('status: ', response.status)
except Exception as ex:
print('Get exception type: ', type(ex).__name__)
return web.Response(text="ok")
async def init():
app = web.Application()
app.router.add_route('GET', '/', func)
return app
web.run_app(init(), port=8111)
and run nginx
to serve 'http://127.0.0.1/index.html'
, the error still appears.
Did you notice after these errors that Nuster did not consume 100% of the CPU? (Bag3)
Do you mean consume 100% cpu
instead of did not consume
?
No, nuster did not consume 100% of the CPU.
from nuster.
Hi, I've being trying to reproduce bug2/3, but still cannot reproduce. I'm closing it now, feel free to reopen it.
BTW bug1 has fixed and merged into master.
thanks
from nuster.
Related Issues (20)
- Updates & deb packages HOT 10
- 实际测试的性能,比作者说的差距非常大。结论:没有必须使用nuster,直接使用nginx就可以。 HOT 3
- How to use Haproxy v2.4.0 HOT 4
- question: how to ignore a specific query parameter for http cache? HOT 5
- Support 304 answers? HOT 2
- Can we force the first request return stale cache when stale on? HOT 7
- Enhancement library improvements for nuster!
- Does `wait on` even work? HOT 4
- CORS HOT 10
- timeout if server directive is missing port HOT 5
- Log cache hit or miss in log file HOT 2
- incorrect date header HOT 2
- backend response headers not passed to client HOT 4
- ARM64 Docker images
- Will caching work for forward proxy? HOT 2
- Equivalent of hash_always_miss HOT 1
- stat `store.disk.cache.loaded` vary depending on running as daemon
- Nuster seems to ignore cache-control headers from BE. Like "no-cache" HOT 5
- will nuster be updated with latest features of haproxy?
- [Security] Infinite loop in src/http_ana.c
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nuster.