genouest / biomaj-watcher Goto Github PK

Hi,

I've installed biomaj-watcher and I've noticed that when I log in fromthe interface of biomaj-watcher and I log out it seems to work correctly.
However, if I press the "reload" button from my web browser, it appears that I'm still logged in :(
Have you any clue of that?

Thx
Emmanuel

Hey guys,

Just dealing with an issue here. I've successfully made a new bank and post-process based off of the ALU example that ships with biomaj3. I've tried to make another post-process that utilizes biomaj2galaxy and I'm having some difficulty. I've taken a few screenshots and combined them to show (on the right) the input that I gave.

It says Invalid Configuration and won't submit. Also, the server gives the following messages:
2015-09-04 13:42:14,276 WARNI [root][waitress] db.formats is not set
2015-09-04 13:42:14,276 WARNI [root][waitress] SMTP mail config not set, you will not be able to send emails

I'm not sure if that's related at all. Also, note how it's showing as "null" for blanks for the sample post-process but just "" for blanks for the post-process I'm trying to add.

So I'm not sure whether that's a bug or something wrong with the way I'm trying to run it. I'd really appreciate some guidance here.

Thanks again!

HI,

I've just started using biomaj-watcher and I found a small issue. If celerery.queue is not defined in global.properties, the application pserver does not start and crahes:

...
  File "/pasteur/services/banques/biomaj/biomaj3/lib/python2.7/site-packages/venusian-1.0-py2.7.egg/venusian/__init__.py", line 205, in scan
    __import__(modname)
  File "/pasteur/services/banques/biomaj/biomaj3/biomaj-watcher/biomajwatcher/background.py", line 7, in <module>
    queue = BiomajConfig.global_config.get('GENERAL', 'celery.queue')
  File "/usr/local/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'celery.queue' in section: 'GENERAL'

Could it be possible to only warn ? I do not intend to use celerey at all with biomaj-watcher.

Emmanuel

If elasticsearch is not used for indexing, the search input form should be hidden in web interface

Path to file is specified both in ini and yaml files, should only use one. Update doc

Hey guys,

So currently I'm trying to add alu to biomaj-watcher. I took alu.properties and put it in the specified etc folder and then I ran biomaj-cli.py --config config path --bank alu --update

The output of that is:
2015-09-16 11:38:44,760 WARNI [root][MainThread] Bank:alu:Update
2015-09-16 11:38:44,760 INFO [root][MainThread] Bank:alu:Release:latest
2015-09-16 11:38:44,761 INFO [root][MainThread] Workflow:Start
2015-09-16 11:38:44,761 INFO [root][MainThread] Workflow:Start:init
2015-09-16 11:38:44,761 INFO [root][MainThread] Workflow:wf_init
2015-09-16 11:38:44,762 INFO [root][MainThread] Workflow:Start:check
2015-09-16 11:38:44,762 INFO [root][MainThread] Workflow:wf_check
2015-09-16 11:38:44,762 INFO [root][MainThread] Workflow:Start:depends
2015-09-16 11:38:44,762 INFO [root][MainThread] Workflow:wf_depends
2015-09-16 11:38:44,762 INFO [root][MainThread] Workflow:wf_depends:True
2015-09-16 11:38:44,763 INFO [root][MainThread] Workflow:Start:preprocess
2015-09-16 11:38:44,763 INFO [root][MainThread] Workflow:wf_preprocess
2015-09-16 11:38:44,763 INFO [root][MainThread] PROC:PRE
2015-09-16 11:38:44,766 INFO [root][MainThread] Workflow:Start:release
2015-09-16 11:38:44,766 INFO [root][MainThread] Workflow:wf_release
2015-09-16 11:38:44,766 INFO [root][MainThread] Workflow:wf_release:previous_session:None
2015-09-16 11:38:44,766 INFO [root][MainThread] Workflow:Start:download
2015-09-16 11:38:44,766 INFO [root][MainThread] Workflow:wf_download
2015-09-16 11:38:45,666 INFO [root][MainThread] Workflow:wf_download:release:remoterelease:2003-11-26
2015-09-16 11:38:45,666 INFO [root][MainThread] Workflow:wf_download:release:release:2003-11-26
2015-09-16 11:38:45,714 INFO [root][Thread-7] Start download thread
2015-09-16 11:38:45,715 INFO [root][Thread-8] Start download thread
2015-09-16 11:38:45,715 INFO [root][Thread-9] Start download thread
2015-09-16 11:38:45,717 INFO [root][Thread-10] Start download thread
2015-09-16 11:38:46,933 INFO [root][MainThread] Workflow:wf_download:Download:Threads:Over
2015-09-16 11:38:46,934 INFO [root][MainThread] Workflow:wf_uncompress
2015-09-16 11:38:46,976 INFO [root][MainThread] Workflow:wf_copy
2015-09-16 11:38:47,000 INFO [root][MainThread] Workflow:wf_copydepends
2015-09-16 11:38:47,001 INFO [root][MainThread] Workflow:Start:postprocess
2015-09-16 11:38:47,001 INFO [root][MainThread] Workflow:wf_postprocess
2015-09-16 11:38:47,002 INFO [root][MainThread] PROC:POST:BLOCK
2015-09-16 11:38:47,002 INFO [root][MainThread] PROC:POST:BLOCK:BLOCK1
2015-09-16 11:38:47,003 INFO [root][Thread-11] PROC:META:RUN:META0
2015-09-16 11:38:47,006 INFO [root][Thread-11] PROC:META:RUN:PROCESS:scanflatdir
2015-09-16 11:38:47,008 INFO [root][Thread-11] PROCESS:RUN:META0_scanflatdir
2015-09-16 11:38:47,183 INFO [root][Thread-11] PROCESS:EXEC:META0_scanflatdir:True
2015-09-16 11:38:47,184 INFO [root][Thread-11] PROC:META:RUN:PROCESS:savetogalaxy
2015-09-16 11:38:47,184 INFO [root][Thread-11] PROCESS:RUN:META0_savetogalaxy
2015-09-16 11:38:47,220 ERROR [root][Thread-11] PROCESS:ERROR:META0_savetogalaxy
2015-09-16 11:38:47,220 INFO [root][Thread-11] PROCESS:EXEC:META0_savetogalaxy:False
2015-09-16 11:38:47,223 ERROR [root][MainThread] Error during task postprocess
2015-09-16 11:38:47,223 INFO [root][MainThread] Workflow:wf_over
2015-09-16 11:38:47,224 INFO [root][MainThread] Notify:mark
2015-09-16 11:38:47,225 INFO [root][MainThread] BANK[alu] - STATUS[False] - UPDATE[True] - REMOVE[False] - RELEASE[2003-11-26]
2015-09-16 11:38:47,326 ERROR [root][MainThread] Could not send email: [Errno 111] Connection refused

And my alu.properties file is:
[GENERAL]
mail.user = 439618f42b537ee5c
protocol = ftp
mail.smtp.host = mailtrap.io
historic.logfile.level = DEBUG
scanflatdir.args = --scan $datadir/$dirversion/$localrelease --type=nucleic --tags="organism:human"
bank.num.threads = 4
savetogalaxy.name = savetogalaxy
elastic_index = biomaj
savetogalaxy.native =
data.stats = 1
savetogalaxy.exe = add_galaxy_library.py
use_ldap = 0
block1.db.post.process = META0
savetogalaxy.desc = test
remote.dir = /blast/db/FASTA/
dir.version = ncbi/blast/alu
log.dir = /home/mark/custombiomaj/log
meta0 = scanflatdir,savetogalaxy
scanflatdir.types =
mail.admin = mark
savetogalaxy.format =
files.num.threads = 4
offline.dir.name = offline/ncbi/blast/alu_tmp
elastic_nodes = localhost
http.group.dir.name = 1
remote.files = ^alu..gz$
process.dir = /home/mark/custombiomaj/process
use_elastic = 0
http.group.file.date = 2
data.dir = /home/mark/custombiomaj/banks
ldap.host = localhost
savetogalaxy.types =
scanflatdir.docker =
blocks = BLOCK1
scanflatdir.name = scanflatdir
db.name = alu
scanflatdir.desc = scan bank flat to detect available files
scanflatdir.files =
scanflatdir.format =
http.group.dir.date = 2
http.parse.dir.line = <a[\s]+href="([\S]+)/".alt="[DIR]">.([\d]{2}-[\w\d]{2,5}-[\d]{4}\s[\d]{2}:[\d]{2})
http.parse.file.line = <a[\s]+href="([\S]+)".([\d]{2}-[\w\d]{2,5}-[\d]{4}\s[\d]{2}:[\d]{2})[\s]+([\d.]+[MKG]{0,1})
scanflatdir.native =
savetogalaxy.files =
celery.broker = mongodb://localhost:27017/biomaj_celery
http.group.file.name = 1
scanflatdir.exe = scan.py
scanflatdir.cluster = false
db.url = mongodb://localhost:27017
visibility.default = public
keep.old.version = 1
celery.queue = biomaj
cache.dir = /home/mark/custombiomaj/cache
savetogalaxy.cluster = false
ldap.dn = nodomain
db.type = nucleic_protein
auto_publish = 0
server = ftp.ncbi.nih.gov
http.group.file.size = 3
mail.password = 01f1e8b4712280
mail.from = biomaj@localhost
db.fullname = alu.n : alu repeat element. alu.a : translation of alu.n repeats
local.files = ^alu.(a|n).*
ldap.port = 389
savetogalaxy.args = -u http://localhost:8080 -k c9bbb8c84e145e285f0b8b82f2cf6cd5 -l "$dbname" --lib-desc "just testing" -f "$dirversion" $datadir/$dirversion/$localrelease
conf.dir = /home/mark/custombiomaj/etc
savetogalaxy.docker =

So alu does show up in biomaj-watcher:

But when I go to edit, the fields are blank:

And when I edit in the fields manually and try to save, I get the following response:

And the console only gives the following message:
2015-09-16 11:54:10,013 WARNI [root][Dummy-1] db.formats is not set

so no error there.

Any ideas?

Thanks!

Hey again,

So one more issue I'm having: when I'm logged in, the Logout button and username button appear as they should. But I never get a Schedule button at the top. I can access that page by typing in the url #/schedule, though.

I've also tried it, removing the custom styling, and it still doesn't show up.

Also one more quick question: how does the Search function work? When I search for alu for example it doesn't return a list of results. Is it meant to search for other things instead?

Thanks again,
Mark.

Hi,

I've installed biomaj-watcher next to biomaj. Biomaj-watcher is complaining about a conflict version of pymongo (3.2 for biomaj, and 2.7.2 for biomaj-watcher).
Could it be possible to change the required version in setup.py to match biomaj?
Thanks

Emmanuel

biomaj provides a FTP server on public banks (or private if authenticated).
Would be nice to get an endpoint that would return a template for a specified bank that could be used by an other biomaj instance to access local ftp server.

Dear biomaj developers,
we try to install biomaj-watcher by following the instructions in README.md

We finally got to the point where we try to start the server. We have edited the development.ini file to point it to our biomaj3 global.properties . At startup pserve raise the following error :

(biomaj-watcher-042017) [banko@banko3 biomaj-watcher]$ pserve - development.ini
Traceback (most recent call last):
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/bin/pserve", line 11, in <module>
    load_entry_point('pyramid', 'console_scripts', 'pserve')()
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/pyramid-1.5.1-py2.7.egg/pyramid/scripts/pserve.py", line 51, in main
    return command.run()
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/pyramid-1.5.1-py2.7.egg/pyramid/scripts/pserve.py", line 316, in run
    global_conf=vars)
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/pyramid-1.5.1-py2.7.egg/pyramid/scripts/pserve.py", line 340, in loadapp
    return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 146, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/biomaj-watcher/biomajwatcher/__init__.py", line 34, in main
    BiomajConfig.load_config(config['biomaj']['config'])
  File "/usr/local/banko/VirtualEnv/biomaj-watcher-042017/lib/python2.7/site-packages/biomaj_core-3.0.7-py2.7.egg/biomaj_core/config.py", line 89, in load_config
    raise Exception('Missing global configuration file')
Exception: Missing global configuration file 

Have we missed a specific configuration step ?
Best regards

Hey guys,

I managed to get BioMAJ and BioMAJ Watcher up and running and I created a new databank, named it alu, and used all the same settings as inside the alu.properties file. It generated a properties file in the right place and everything seemed to be working:

Clicking on alu opens the bank properties as expected:

Now clicking on the little eye next to Release: 2003-11-26 gets me to this page:

Where it seems none of the strings are showing up properly. The back button also doesn't work. Clicking on show format files also gives me similar results:

So, any ideas on how to fix?

Thanks!

Hello,
The table shown in BioMAJ Watcher does not indicate the date of the latest bank download.
This is an important information for any user who whishes to evaluate the freshness of the data available on a site. Is this information available somewhere else ?
Best regards,
Olivier

Hi Biomaj team,

I wonder if you had this issue before:
I installed Biomaj3 on a centos 6.6 machine and it's working (I checked) and then I installed Biomaj-wtahcer and launched it (just changing the admin, port and globa_properties parameters in the production.ini file).
There is no apparent error messages and I can access to the web interface but when I click on the different buttons nothing happen (it seems like the interface is not linked with the background). Have you had any similar issue before? Do you have an idea of what could have gone wrong?

Thank you very much.

Hey guys,

So I've spent a bit of time and installed biomaj and biomaj-watcher and all their dependencies (mongodb, python venv, etc) and finally got to a point where the server starts properly. I made sure to refer to my global.properties in the seed.py and development.ini.

So when I turn my web browser to the specified IP (where it says serving on http://x.x.x.x:xxxx), it opens a white page saying 404 Not Found - The resource could not be found: http:/0.0.0.0:6543/app/

So I'm just wondering if I missed any steps and if so what do I need to do to get it working?

Thanks

Hey guys,

I'm currently setting up BioMAJ and BioMAJ-Watcher in a docker container. I've got it running and everything so far, but when I try to create a new bank, I get the following error:

2015-09-17 20:01:55,872 DEBUG [root][waitress] Initialize alu
2015-09-17 20:01:55,872 ERROR [root][waitress] Bank configuration file does not exists
2015-09-17 20:01:55,873 ERROR [waitress][waitress] Exception when serving /bank/alu/config
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.10-py2.7.egg/waitress/channel.py", line 336, in service
task.service()
File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.10-py2.7.egg/waitress/task.py", line 169, in service
self.execute()
File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.10-py2.7.egg/waitress/task.py", line 388, in execute
app_iter = self.channel.server.application(env, start_response)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/router.py", line 242, in call
response = self.invoke_subrequest(request, use_tweens=True)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/router.py", line 217, in invoke_subrequest
response = handle_request(request)
File "/usr/local/lib/python2.7/dist-packages/pyramid_debugtoolbar-2.4.1-py2.7.egg/pyramid_debugtoolbar/toolbar.py", line 178, in toolbar_tween
return handler(request)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/router.py", line 163, in handle_request
response = view_callable(context, request)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 596, in call
return view(context, request)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 329, in attr_view
return view(context, request)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 305, in predicate_wrapper
return view(context, request)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 355, in rendered_view
result = view(context, request)
File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 501, in _requestonly_view
response = view(request)
File "/home/bio/biomaj/biomaj-watcher/biomajwatcher/views.py", line 484, in update_bank_config
bank = Bank(name, no_log=True)
File "/usr/local/lib/python2.7/dist-packages/biomaj-3.0.7-py2.7.egg/biomaj/bank.py", line 53, in init
self.config = BiomajConfig(self.name, options)
File "/usr/local/lib/python2.7/dist-packages/biomaj-3.0.7-py2.7.egg/biomaj/config.py", line 129, in init
raise Exception('Configuration file '+bank+'.properties does not exists')
Exception: Configuration file alu.properties does not exists

I input the information in the fields exactly the same as on the host machine, where it saves properly. But it refuses to save in the docker and gives that error in the console.

If it helps diagnose potential installation issues, the docker project can be seen here: https://github.com/markiskander/biomaj-docker

Thanks again!