Discussion: Integrate our fork about vmprof-server HOT 28 OPEN

Cool, we are through with the internal open source process, I'll release the source by the end of the week.

from vmprof-server.

cc @StephanErb

from vmprof-server.

Hi, looks like a solid enhancement to vmprof-server. As I see things now: I would stand up for possibilty B. We already maintain and run the service and this will continue that way.

How did you solve the SQL storage issue? Store the gzipped profiles on the file system? If you ask me we should not continue to store json in the relational database. For jitviewer the gzipped file is stored on the file system which is good enough as far as I can tell.

from vmprof-server.

In any case, is the source available on the web? I would to look at it for a bit to estimate how much work it would be to merge it. Or would you just open a pull request then?

from vmprof-server.

How did you solve the SQL storage issue? Store the gzipped profiles on the file system?

Yes

In any case, is the source available on the web? I would to look at it for a bit to estimate how much work it would be to merge it. Or would you just open a pull request then?

As I said I can't make it available at the moment but I'll send you a private, confidential copy in a few seconds.

from vmprof-server.

Any news? I'm planning to release a bug fix version today, so we could aim for the next major release

from vmprof-server.

Have you had a look at the code? I haven't put in the effort of making our server open source yet; I expected your "complexity of integration assessment" first. But we can also make it open source first.

from vmprof-server.

Yes I did, nothing that takes tremendous amount of re-engineering. If you can make it open source, we will integrate it into the public service. After all this is a feature many people want to have!

from vmprof-server.

Here we go!

https://github.com/blue-yonder/vmprof-viewer-client
https://github.com/blue-yonder/vmprof-viewer-server

from vmprof-server.

Great! I assume that the licence of both is compatible to MIT? I'll probably find some time soon to pull in your fork!

from vmprof-server.

Awesome! I'm happy to assist. License is MIT, yes. The features that are most important for us are memory profiles with good stack traces and higher-performance profile storage (not in the SQL database)

from vmprof-server.

What's the current state of this? Is there anything I can do to help get the merge done?

from vmprof-server.

There are a view commits I have made, the migrations are ready but I need to apply some changes here and there (model names have changed, ...). We also need to copy and modify the client in the vmprof package. That would be a small project on it's own.

from vmprof-server.

OK, let me know if when should help out.

from vmprof-server.

I'm planning to work on vmprof this friday (Munich, pycon.de) and maybe I find time to push this forward. It would be great if you have a look on integrating the client side code to github.com/vmprof/vmprof-python

from vmprof-server.

@jonashaag , yesterda. I set up vmprof-viewer-server according as https://github.com/blue-yonder/vmprof-viewer-server

but meet a mistake:
(env) [root@localhost vmprof-viewer-server]# vmprof_viewer/manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
November 22, 2016 - 03:03:11
Django version 1.10.3, using settings 'vmprof_viewer.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Unhandled exception in thread started by <function wrapper at 0x31c92a8>
Traceback (most recent call last):
File "/root/vmprof-viewer-server/env/lib/python2.7/site-packages/django/utils/autoreload.py", line 226, in wrapper
fn(*args, **kwargs)
File "/root/vmprof-viewer-server/env/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 142, in inner_run
handler = self.get_handler(*args, **options)
File "/root/vmprof-viewer-server/env/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/runserver.py", line 27, in get_handler
handler = super(Command, self).get_handler(*args, **options)
File "/root/vmprof-viewer-server/env/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 64, in get_handler
return get_internal_wsgi_application()
File "/root/vmprof-viewer-server/env/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 59, in get_internal_wsgi_application
sys.exc_info()[2])
File "/root/vmprof-viewer-server/env/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 49, in get_internal_wsgi_application
return import_string(app_path)
File "/root/vmprof-viewer-server/env/lib/python2.7/site-packages/django/utils/module_loading.py", line 20, in import_string
module = import_module(module_path)
File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module
import(name)
django.core.exceptions.ImproperlyConfigured: WSGI application 'vmprof_viewer.wsgi.application' could not be loaded; Error importing module: 'No module named wsgi'

What can I do to fix it ?

from vmprof-server.

Create a file vmprof_viewer/wsgi.py with the following contents

"""
WSGI config for myproject project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/1.7/howto/deployment/wsgi/
"""

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "vmprof_viewer.settings")

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

from vmprof-server.

I have been working on the server to display the memory graph. Since we agreed to not have separate files, I wonder what the file format for the mem.msgpack.gz and addr_name_map.msgpack.gz is.

What I'm currently trying to do is reconstruct mem and addr_name_map on ther server from the vmprof profile.

from vmprof-server.

IIRC:

mem: List of [ip, mem bytes] (see https://github.com/blue-yonder/vmprof-viewer-client/blob/master/vmprof_viewer_client/protocol.py#L34)

addr_name_map: Dict of ip -> string name of function (see https://github.com/blue-yonder/vmprof-viewer-client/blob/master/vmprof_viewer_client/protocol.py#L40)

from vmprof-server.

the resampling is now again done on the server (as shortly discussed in the commit comment), I got most of it working I think. That is missing now on master (just to summarize):

• peak memory is not displayed (top right)
• duration is not displayed (top right)
• relative and absolute time is not yet working
• profiles cannot be grouped to projects
• function profiling

for the last point (function profiling) I think there is some special decorator, can you maybe point me to an API or example how it should be used? I'm unsure how top_level_function name is sent to the server.

from vmprof-server.

Cool! We can leave out the function profiling stuff and grouping for now. Function profiling is something that can also live in a separate project; it's a mere convenience wrapper.

from vmprof-server.

Hi,
as Jonas is no longer with us, I was assigned to this project.
I think that #11 (comment) is still an accurate summary as of now.
What can we do in order to get the missing features into the server? Maybe without function profiling, I guess we could provide that externally, and if you're interested, integrate that later.

from vmprof-server.

Sorry for the late response! I think a first step it would be good to test the current setup as it is. One problem was and is that like the flamegraph visualisation there is no documentation about what it means. E.g. what is the difference between absolute and relative in the memory view (where is the origin of 'relative').

Here are some issues I remember:

• we use PyPy for generating the output sent to the browser (that contains the memory view data), it sometimes occurs that a numpy array is resized which is not supported on pypy
• Absolute/Relative button are not working

I have been thinking to extend vmprof.com to make a short tutorial as an overlay to the flamegraph that shows the essential details (as it is done in the jitlog). I think that would be a good addition to the memory view as well.

from vmprof-server.

Hi guys, if you have any questions, I'm happy to help.

Flamegraph: Not sure what you mean. The memory graph should be pretty obvious.

@Corni most of the differences between our internal vmprof frontend and the vmprof.com frontend should be easy to add, except for the grouping/project structure. Not sure if you guys @planrich are interested in that at all?

from vmprof-server.

'pretty obvious' is a stretchable term. at least I have experienced that some people have no clue what the flame graph means. Usually you get no feedback at all and from time to time you find out that they kind of guess how it "should" be (profiling is not about how it should be, but how it is IMHO). So my idea is to make the docs easily accessible (preferable in the same view as the profiling visualisation).

from vmprof-server.

Flame graph = the blue line? I'm confused by the term "flame graph" here... it's a completely different kind of visualisation that the CPU flame graph.

If flame graph = blue line, do you recall what confuses people about it?

from vmprof-server.

no, I was taking about the CPU visualisation (=>CPU flame graph view). What confused me about the memory view is relative/absolute. does relative mean, "relative" to the minium heap size (= subtract the minimum heap size)?

from vmprof-server.

Um, it was relative/absolute TIME. The reasoning here was that some people know "this must have been somewhen around 11:30 yesterday" and other people look for "about 20 minutes into the program run".

from vmprof-server.