Scenario:

$ pip install invenio-base
$ inveniomanage instance create myapp

Result:

Traceback (most recent call last):
  File "/Users/nharraud/.virtualenvs/b2share-scratch/bin/inveniomanage", line 7, in <module>
    from invenio_base.manage import main
  File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/manage.py", line 25, in <module>
    from .factory import create_app
  File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/factory.py", line 47, in <module>
    from .wrappers import Flask
  File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/wrappers.py", line 75, in <module>
    from .signals import before_handle_user_exception
  File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/signals.py", line 26, in <module>
    from blinker import Namespace
ImportError: No module named blinker

• introduce new documentation structure inveniosoftware/cookiecutter-invenio-module#56
• document user perspective
• document signals (if not done)
• document example app (if not done)

The current locations of instance and folder paths doesn't work nicely when used through python manage.py.

The package contains examples/app.py example application, but there is no test
for it in tests/test_example_app.py. It should be added. See existing examples:

• https://github.com/inveniosoftware/invenio-records/blob/master/tests/test_example_app.py
• https://github.com/inveniosoftware/invenio-marc21/blob/master/tests/test_example_app.py

Problem

I've got a module for registering a "draft" version for each record found in INVENIO_RECORDS_REST. I'd like to have the draft endpoints to behave exactly the same way as normal invenio record rest endpoints - for example to be able to use url_for('invenio_records_rest.*_item') and use all the infrastructure created around records.

The only working solution that I've been able to find so far is to, at some point, add those extra records into INVENIO_RECORDS_REST.

This must be done before records_rest blueprint is created, i.e. before the blueprint loader is invoked in _create_app.

Currently there is no signal called after extensions are loaded (this is the moment when merged config becomes available) but before blueprint loading.

Workaround

I was able to find a workaround - in my extension's init_app I replace INVENIO_RECORDS_REST dictionary with an instance of my own class that creates draft endpoints when getters (such as .items()) are invoked, but this hack is too fragile as it might be called prematurely by external module trying to access rest configuration.

Suggestion

I'd suggest adding the following extra signals that would work in cooperation with already existing app_created/app_loaded called in this order:

• app_config_loaded - called just after loading config via config loader

• app_converters_loaded - called here when converters are loaded

• app_extensions_loaded - called after extensions are loaded and before blueprints are created

The new signals would have the same signature as the existing ones, that is app_*_loaded(_create_app, app=app)

The signals could be used in various cases when one would like to modify the app before the next phase of loading.

Alternatives

For my use case the workaround above currently works. Another alternative would be to copy/paste/adapt invenio startup
sequence to include the modification code and call this instead of invenio's default.

Investigate if it is not better to distribute the cookiecutter template separately of Invenio-Base.

Creating a new Invenio instance documentation is wrong. The command actually outputs:

$ inveniomanage instance create test
/Users/lnielsen/src/invenio-base/invenio_base/cli.py:45: UserWarning: This command have been deprecated.
  warnings.warn('This command have been deprecated.')
This command have been deprecated. Please use this instead:

pip install cookiecutter
cookiecutter https://github.com/inveniosoftware/cookiecutter-invenio-instance

Note, there's also a spelling mistake above which should also be fixed: This command have been -> This command has been.

Probably removing the section is the best, since this should be documented in general Invenio documentation instead.

there are two deprecation warnings concerning ProxyFix class:

tests/test_wsgi.py::test_proxyfix[2-4.3.2.1]
  /Users/janek/.virtualenvs/invenio-files/lib/python3.6/site-packages/werkzeug/middleware/proxy_fix.py:88: DeprecationWarning: 'num_proxies' is deprecated as of version 0.15 and will be removed in version 1.0. Use 'x_for' instead.
    self.num_proxies = num_proxies

and

/Users/janek/invenio-software/src/invenio-base/invenio_base/wsgi.py:53: DeprecationWarning: 'werkzeug.contrib.fixers.ProxyFix' has moved to 'werkzeug.middleware.proxy_fix.ProxyFix'. This import is deprecated as of version 0.15 and will be removed in 1.0.
    return ProxyFix(wsgi_app, num_proxies=app.config['WSGI_PROXIES']) 

The two CLI commands below are not documented:

inveniomanage instance entrypoints
inveniomanage instance migrate-secret-key

Especially secret key migration would need docs - i.e. how should other modules hook into the secret key migration.

Check also API documentation for missing modules.

There doesn't seem to be any way to disable the /static endpoint generated for each application in invenio_base.app:base_app. When static_folder is set to None (default value), it generates a path to the instance directory.
This creates an /api/static endpoint.

Invenio-base wsgi.py should be updated for new werkzeug

invenio_base/wsgi.py:36: DeprecationWarning: 'werkzeug.wsgi.DispatcherMiddleware' has moved to 'werkzeug.middleware.dispatcher.DispatcherMiddleware'

The Werkzeug v2.3.0 release, removed many deprecations, some of which are used by or break Invenio modules and their dependencies. More notably so far we've discovered:

• Authorization headers with Bearer tokens, failing to load in Invenio-OAuth2Server

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in full_dispatch_request
    rv = self.preprocess_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2313, in preprocess_request
    rv = self.ensure_sync(before_func)()
  File "/usr/local/lib/python3.9/site-packages/invenio_oauth2server/ext.py", line 189, in verify_oauth_token_and_set_current_user
    valid, req = oauth2.verify_request(scopes)
  File "/usr/local/lib/python3.9/site-packages/flask_oauthlib/provider/oauth2.py", line 529, in verify_request
    return self.server.verify_request(
  File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/base.py", line 64, in wrapper
    return f(endpoint, uri, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/resource.py", line 69, in verify_request
    request.token_type = self.find_token_type(request)
  File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/resource.py", line 85, in find_token_type
    estimates = sorted(((t.estimate_type(request), n)
  File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/resource.py", line 85, in <genexpr>
    estimates = sorted(((t.estimate_type(request), n)
  File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/tokens.py", line 300, in estimate_type
    if request.headers.get('Authorization', '').startswith('Bearer'):
TypeError: 'NoneType' object is not callable

For the above reasons we have currently pinned Werkzeug to <2.3.0.

In Flask v2.2 before_(app_)first_request was deprecated and will be removed in v2.3.

We are currently using this through various modules for registering menus or other special cases:

• before_first_request
• before_app_first_request
• (Optional, but similar) Blueprint.record_once (see #171 (comment) 👇 )

We should either:

• Pin Flask < 2.3 in this module
• Refactor the code not to use these decorators. This is pretty difficult since the proposed alternative is to "Run setup code when creating the application instead."... This means we have to hook in the invenio-app lifecycle to introduce a similar mechanism.

• check existing API functionality
• add missing important API functionality
• check API function signatures and parameters
• enhance API docstrings (param, returns, raises, versionadded)
• plug API functions to existing docs

Currently we have the following environments where we need a Flask application running:

• Celery
• WSGI
• CLI

For all of them we currently create two Flask application factories:

• UI app (create_app)
• API app (UI app creates it and adds it to /api)

Many core Flask Extensions are initialised twice - once on the UI app and once on the API app. The problem with this approach is in all scenarios we create two Flask applications even if we don't need them. E.g. the Celery application only needs one Flask application. Also the CLI shell command and many other commands would need just one Application. The CLI run command will however need the full WSGI application (i.e. the two apps) to run.

It seems to me that our current approach is not ideal, but however also don't have any other ideal solutions.

• check existing API functionality
• add missing important API functionality
• check API function signatures and parameters
• enhance API docstrings (param, returns, raises, versionadded)
• plug API functions to existing docs

https://www.quantifiedcode.com/app/project/gh:inveniosoftware:invenio-base/diff/26fd70fb4f34cb1446ba56e08a33628961bd68ea/7dac34bf68a3bd434e8bd75a59cfb7cd716b7b5c/issues?groups=code_patterns%3A1sMHNVxe%3Af0,code_patterns%3A539Wia7V%3Af0,code_patterns%3A4ACGxFj1

Package version (if known): V12 latest

Describe the bug

After changing the SECRET_KEY in invenio.cfg and running invenio instance migrate-secret-key --old-key CHANGE_ME, users are unable to log in with the message "incorrect password". It appears the command accepts any value as old-key without regenerating the secrets with the new secret key, posing a potential security risk to not be able to access the system after.

Steps to Reproduce

1. Create a user using user/password method
2. Make sure you able to login with your password.
3. Change SECRET_KEY in invenio.cfg.
4. Run the command invenio instance migrate-secret-key --old-key CHANGE_ME.
5. Attempt to log in with a user account using username and password.
6. Encounter "incorrect password" error.

Expected behavior

Users should be able to log in with their credentials after the secret key migration, and the process should validate the old-key properly to ensure secrets are regenerated with the new secret key securely.

Additional context

The issue was discovered during testing of the migrate-secret-key command, raising concerns about the command's validation of the old-key parameter and the regeneration process of secrets.

Same as:

• invenio-records-ui #84
• invenio-records-rest #289
• invenio-search-ui #73
• invenio-previewer #111
• invenio-theme #145
• invenio-base #147
• invenio-i18n #81

https://travis-ci.org/inveniosoftware/invenio-base/jobs/656676729

Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/travis/build/inveniosoftware/invenio-base/tests/test_wsgi.py", line 109, in appview
    return jsonify(data)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/json.py", line 254, in jsonify
    if current_app.config['JSONIFY_PRETTYPRINT_REGULAR'] and not request.is_xhr:
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
AttributeError: 'Request' object has no attribute 'is_xhr'

Fix also deprecation warning in Travis config:
https://travis-ci.org/inveniosoftware/invenio-base/jobs/656676729/config

Check if actually needed, or we remove it.

Change to e.g. depend on e.g. invenio[base,auth,metadata]. Add docker support? Better documentation?

As discussed:

• modules clean-up + ensure installation is working fine
• define how db/es different versions are specified
• define strategy/protocol to upgrade
• dependencies checker
• setup recurrent test run/cron (run docker on Travis?)
• reusable docker

• ask which endpoint should be registered as /
• otherwise create view with Welcome to {{ site_name }}!

flask-shell-ipython is a flask module that changes the default python shell to IPython when running the command flask shell (or invenio shell in our case)

Improve naming convention of ext_entry_points and bp_entry_points (see https://github.com/inveniosoftware/invenio-base/blob/master/invenio_base/app.py#L42-L43)

• rename arguments
• update cookiecutter templates

cc @lnielsen

See comments in #31

conf_loader
bp_entry_points
ext_entry_points

• Add integration tests to inveniosoftware/invenio
• Selenium tests + API tests.

• invenio-base
• invenio-config

As cookiecutter is now based on invenio-app I think that celery.py and wsgi.py should be removed. In any case these two files are broken as they includes factory.py which has been removed.

Needed for inveniosoftware/invenio-app#66

The following cookiecutter change:

inveniosoftware/cookiecutter-invenio-module#98

should be propagated to this Invenio module.

Namely, in run-tests.sh, the sphinx for doctests is invoked after pytest run:

$ tail -3 ./\{\{\ cookiecutter.project_shortname\ \}\}/run-tests.sh
sphinx-build -qnNW docs docs/_build/html && python setup.py test && sphinx-build -qnNW -b doctest docs docs/_build/doctest

This sometimes led to problems on Travis CI with the second sphinx-build run due
to "disappearing" dependencies after the example application was tested.

A solution that worked for invenio-marc21 (see
inveniosoftware/invenio-marc21#49 (comment))
and that was integrated in cookiecutter (see
inveniosoftware/cookiecutter-invenio-module#98) was to
run doctest execution in pytest, removing the second sphinx-build invocation.

This both solved Travis CI build failures and simplified test suite execution.

Note that this change may necessitate to amend the code tests etc so that things
would be executed with the Flask application context (see
inveniosoftware/invenio-marc21@09e98fc).

It might actually be possible to remove since the only really blocking module we found in terms of relying on application context is invenio-pages (because it performs an SQL query to fetch the available pages from the database).

Originally posted by @slint in #172 (comment)

Do we want to use entry_points to load Flask commands? If yes, shall we use entry_points group flask.commands or create new configuration option for the entry_point group name in create_cli factory?

cc @inveniosoftware/triagers

https://travis-ci.org/inveniosoftware/invenio-base

Remove manage.py script from templates and rely only on installed cli.

Further reading http://click.pocoo.org/5/setuptools/

Problem:
It should be possible to update the SECRET_KEY if it is compromized.

Design TBD

Only devel build is breaking

I tried following the instructions to set up a test site:

conda create -n test-invenio-base python=2.7
source activate test-invenio-base
cd invenio-base/  # (git clone of dev version)
pip install .
cd ../
inveniomanage instance create mysite
cd mysite
ls
pip install .

But the mysite command don't exist following this (the docs suggest that mysite --help should work, but I get mysite: command not found. Am I doing something wrong?

Note that doing python setup.py install instead of pip install . doesn't change anything.

Output of pip freeze if needed:

alembic==0.9.2
amqp==2.1.4
angular-gettext-babel==0.3
arrow==0.10.0
Babel==2.4.0
billiard==3.5.0.2
binaryornot==0.4.3
blinker==1.4
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
cookiecutter==1.5.1
dojson==1.3.1
elasticsearch==2.4.1
elasticsearch-dsl==2.2.0
enum34==1.1.6
Flask==0.12.2
Flask-Alembic==2.0.1
Flask-Assets==0.12
Flask-BabelEx==0.9.3
Flask-Breadcrumbs==0.4.0
Flask-CeleryExt==0.3.0
Flask-Collect==1.2.2
Flask-Limiter==0.9.4
Flask-Menu==0.5.1
Flask-SQLAlchemy==2.2
functools32==3.2.3.post2
future==0.16.0
idna==2.5
invenio-app==1.0.0a1
invenio-assets==1.0.0b6
invenio-base==1.0.0a15.dev20160823
invenio-config==1.0.0b3
invenio-db==1.0.0b7
invenio-i18n==1.0.0b3
invenio-indexer==1.0.0a9
invenio-jsonschemas==1.0.0a4
invenio-marc21==1.0.0a5
invenio-oaiserver==1.0.0a12
invenio-pidstore==1.0.0b1
invenio-query-parser==0.6.0
invenio-records==1.0.0b1
invenio-records-rest==1.0.0a18
invenio-records-ui==1.0.0a9
invenio-rest==1.0.0a10
invenio-search==1.0.0a9
invenio-search-ui==1.0.0a6
invenio-theme==1.0.0b2
itsdangerous==0.24
Jinja2==2.9.6
jinja2-time==0.2.0
jsmin==2.2.2
jsonpatch==1.15
jsonpointer==1.10
jsonref==0.1
jsonresolver==0.2.1
jsonschema==2.6.0
kombu==4.0.2
limits==1.2.1
lxml==3.8.0
Mako==1.0.6
MarkupSafe==1.0
marshmallow==2.13.5
mysite2==0.1.0.dev0
node-semver==0.1.1
ordereddict==1.1
pluggy==0.4.0
poyo==0.4.1
pyPEG2==2.15.2
pyrtlsdr==0.2.3
python-dateutil==2.6.0
python-editor==1.0.3
pytz==2017.2
requests==2.17.3
simplejson==3.10.0
six==1.10.0
speaklater==1.3
SQLAlchemy==1.1.10
SQLAlchemy-Utils==0.32.14
urllib3==1.21.1
vine==1.1.3
webargs==1.7.0
webassets==0.12.1
Werkzeug==0.12.2
whichcraft==0.4.1

Package version (if known): v1.2.12

Describe the bug

Remove werkzeug upper pin from here when flask-login fixes the incompatibility.

This task should be built on top of PR #27 (which we can hopefully integrate later today)

Add a command startproject (name TBC) to inveniomanage that will generate an initial project structure using cookiecutter

Example of usage:
inveniomanage startproject mysite

will generate something a project structure like this:
https://github.com/lnielsen/mysite

The project structure should be generated using cookiecutter Pythonic API.

The cookiecutter template should be included in invenio-base and full path to found using pkg_resources.

In order to avoid patterns like this, it would be good if we here can override the url_prefix via a configuration variable instead.

Co-authored-by: @jirikuncar

https://travis-ci.org/inveniosoftware/invenio-base/jobs/560026524