3scale / apicast Goto Github PK
View Code? Open in Web Editor NEW3scale API Gateway
License: Apache License 2.0
3scale API Gateway
License: Apache License 2.0
@jmprusi . Any reason to not merge the https://github.com/3scale/docker-gateway/tree/openshift branch into Master?
NOTE: If you do so, please do early on Tuesday and let me know, as I will need to change the tutorial to refer to master branch content.
Currently the GET /config
call to the management API returns a serialized JSON with text/plain
content type. It would be better to return it as application/json
.
Currently the downloaded config from MT API includes a provider_key in the JSON, and this is used by apicast for debugging calls/headers.
To reduce possible leakage of the global provider_key the proposal is that this key is not used (there will be a separate issue in System dependant on this one, to stop sending it) and the provider_key or access_token that is used in the deploy of the gateway (the one used to fetch the config from MT API in the first place) is used in it's place.
I will assign to the RHAMP 2.0 release milestone, as the MT part of this is breaking the API (JSON file format) - and we want to reduce the legacy versions out there that will need to be updated to this newer version of apicast.
Would be nice to get some feedback on how we should improve logging.
From debugging and troubleshooting to logging request times.
setting THREESCALE_CONFIG_FILE="" causes the container to fail due bad env checks.
This makes errors difficult to debug, as the two different causes are not discriminated.
Return 429 "Too many requests"
There maybe documentation pages that will required modification to explain this code has changed in this (insert version of apicast that implements this here) or apicast
I believe that this is when Service Management API returns a 409 Conflict with authorizing.
We do not at the moment contemplate the need to customize this code and error message via Admin Portal - so a fixed code and message are all that's requested in this issue.
In apicast.conf there is a client_secret_matches location block that is not currently used or needed:
Add a short description of this repo
Add MIT License file
Add the APIcast 2.0 OpenShift template to the Ansible-Openshift installer, so when the customer installs OpenShift, the APIcast templates gets automatically installed.
The GitHub repository is this:
https://github.com/openshift/openshift-ansible
Process
From @sdodson: We just sync the file from your upstream repo into the installer repo. Your teams can either open pull requests against openshift-ansible or email me and let me know the new tag to
include and we'll update it then.
We should output the error message from this part:
https://github.com/3scale/docker-gateway/blob/v2/src/configuration.lua#L232-L241
We should get a more descriptive message like this:
2016/09/07 22:24:40 [error] 24#24: *1 lua entry thread aborted: runtime error:
/opt/app/src/configuration.lua:243: no resolver defined to resolve "3scaletesting-admin.3scale.net"
instead of:
2016/09/07 22:16:22 [error] 23#23: *1 lua entry thread aborted: runtime error: /opt/app/src/configuration.lua:193: missing configuration
While getting apicast running and integrated with 3scale API Management Platform a number of issues can arise that are not familiar or easy to debug for new users. We wish to make this start-up and integration process as easy and painless as possible for new users.
To do so, we would like to add to this upstream project a document where possible sources of problems are documented as a resource for people to use to help them get going.
I suggest adding in doc folder and linking from the README.
This would be an upstream document, focused on apicast alone, based on learnings of 3scale over the years captured in this product document.
https://support.3scale.net/docs/api-devops/troubleshooting
When resolving issues, or finding new causes, anyone can PR this document to add cases that may help other users int he community solve their problems.
2016/09/06 23:32:53 [error] 23#23: *1 lua ssl certificate verify error: (20: unable to get local issuer certificate), client: 172.17.0.4, server: _, request: "GET / HTTP/1.1", host: "testing.com"
2016/09/06 23:32:53 [error] 23#23: *1 lua entry thread aborted: runtime error: /opt/app/src/configuration.lua:186: missing configuration
stack traceback:
coroutine 0:
[C]: in function 'error'
/opt/app/src/configuration.lua:186: in function 'boot'
access_by_lua(apicast.conf:52):5: in function <access_by_lua(apicast.conf:52):1>, client: 172.17.0.4, server: _, request: "GET / HTTP/1.1", host: "testing.com"
Currently if you add in the provider key and admin portal, the nginx configuration will be downloaded for ALL services.
Given that we now have admin rights by service which allow us to cherry pick which configurations get downloaded based on the user rights, would it make sense to update the docker image to use access_tokens or provider_key to download the relevant configs.
Any request sent to the gateway will return 404 unless the Host header is set to "localhost" even when the service has been configured with "testing.com".
nginx fails to read the conf file with access_log
set to /dev/stdout
:latest
nginx starts but does not forward any requests to the private URL
nginx forwards requests
docker run --detach --publish=127.0.0.1:319:8080 --name=3scale --env=THREESCALE_PROVIDER_KEY=$THREESCALE_PROVIDER_KEY --env=THREESCALE_ADMIN_URL=https://tripgo-admin.3scale.net quay.io/3scale/gateway
2016/09/13 05:48:17 [emerg] 7#0: open() "/dev/stdout" failed (13: Permission denied)
maybe this one in /entrypoint.sh: moby/moby#6880 (comment) then point nginx to log to /tmp/logpipe
Currently the access_token
for the calls using OAuth flow is taken:
from the query parameters (for GET) or body (for POST and other methods), in case the service is configured in 3scale to use headers for credentials locations configured.
from access_token
header
While 1 is OK, 2 is not compliant with the RFC 6750, according to which the access_token
should be taken from the Authorization
header:
Authorization: Bearer xxyyzz
This should be fixed.
Another question is whether APIcast should actually check for both locations (thus disregarding the setting in the 3scale configuration) so that it can supports any OAuth client.
Add a basic description what APIcast is to the top of the readme so visitors know what this repository is about.
related to #48
There should be an OpenShift template for deploying the nginx gateway on OpenShift with a service supporting https. The key and certificate for the ssl config come from the volume of an OpenShift secret.
The docker gateway works great with one service.
If we have multiple service support, the gateway always routes to call to first server block on nginx conf file and then to upstream backend.
Below is the file format 3scale generates
http {
upstream backend_abc {
...
}
server {
server_name $hostname;
location /{
set $service_id 2555555;
...
...
}
}
upstream backend_def {
...
}
server {
server_name $hostname;
location /{
set $service_id 2444444;
...
...
}
}
upstream backend_xyz {
...
}
server {
server_name $hostname;
location /{
set $service_id 233333;
...
...
}
}
}
Since the hostname always remain the same, the call always get routed to service_id 2555555 of the first server block and ultimately to upstream backend_abc.
Let me know if you need more information.
service.api_backend
normally (when created with UI) includes the port (i.e. https://echo-api.3scale.net:443
), however, when the port is not specified, it will not be set here, and as a result the balancer will fall back to the port 80
, even if the schema is https
. This causes a 502 Bad Gateway
error with the following in the error log:
2016/12/22 11:12:17 [error] 52363#0: *5 SSL_do_handshake() failed (SSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) while SSL handshaking to upstream, client: 127.0.0.1, server: _, request: "GET /test?user_key=asdf HTTP/1.1", upstream: "https://52.54.57.126:80/test?user_key=asdf", host: "echo-api.3scale.net"
To reproduce, take a working configuration with valid credentials and set api_backend
to https://echo-api.3scale.net
:
{
"id": 1234567890987,
"provider_key": "provider-key",
"services": [
{
"id": 654321,
"backend_version": "1",
"backend_authentication_type": "service_token",
"backend_authentication_value": "service-token-1234567890",
"proxy": {
"api_backend": "https://echo-api.3scale.net",
"auth_app_key": "app_key",
"auth_app_id": "app_id",
For 3scale backend this is solved this way.
For API backend, we could follow the same way.
Or, alternatively, we could change this line to:
local peers = balancer:peers(ngx.ctx[ngx.var.proxy_host], ngx.var.proxy_port)
proxy_port
will be set to the protocol's default port. Custom ports should still work, because of this precedence, but maybe I'm missing something.
We need to have an option to disable lua code cache and enable code reloading.
That will also make configuration reloading work. Later we might need two different options.
One for code and second for configuration.
[provide output of the nginx -V
or openresty -V
command from openshift/local terminal]
[provide timestamp of the docker image from docker inspect --format='{{.Created}}' quay.io/3scale/apicast:v2
]
bin/apicast -C
does not reload the configuration or code
should reload the configuration and code from filesystem/api
the -C
option is just suggestion. can be something else.
The schema for importing data states that service ids are integers, but they are opaque strings for all intents and purposes within Apicast, and that is bad because we end up converting them everywhere.
There are two ways to fix this. One is changing this line in the configuration schema so it says string
instead of integer
, and changing the configuration generation as needed.
The other way is using this schema but making sure it is loaded with the converted service id at initialization time.
I think this should be considered before making the schema final.
We need an easy way to demo OAuth2 integration of APIcast. Not only demo, but to try in a browser after making changes to verify all works.
Everyone having the necessary software installed should be able to go through full OAuth flow on their local machine.
A production ready version of this:
https://support.3scale.net/codehub/use-prebuilt-cache
Should check the cache for dynamically added keys/tokens - rather than hard code them in whitelist.lua
Should avoid the long in-band wait associated with a 503 with the current solution. Background job?
We wish to make it easier for people new to apicast (who maybe also new to nginx, openresty and lua) to get started, use it and add their customizations.
Here is a list of customization examples that we would like to have explained briefly how to do.
These could be short documents on each one, in the corresponding examples sub-folder or small sections in an overall "customizations" document in examples root, to accompany any code examples for the same in the examples folder
[provide a description of the issue]
I have an issue, first seen in Openshift, where endpoints in the 2nd service I declare aren't picked up - despite setting the host header.
It's not Openshift specific, I've setup a 3scale AMI which you can access and API requests routed through it see the same issue.
ssh -i ./_3scale-us-east.pem [email protected] (pem is available to the 3scale dev team by emailing me - actually it's on an earlier email)
this restart commend give you the location of the configs:
sudo /opt/openresty/nginx/sbin/nginx -p /opt/openresty/nginx/ -c /opt/openresty/nginx/conf/nginx.conf -s reload
I've uploaded the auto-gen'd config files - zero modifications.
NOTE: I have added this to the Integration page in 3scale as suggested:
Public Base URL http://ec2-107-22-18-40.compute-1.amazonaws.com:80
See JSON
https://cathay-poc-os-admin.3scale.net/admin/api/nginx/spec.json
By contrast, the other call in the other service (flights) works:
https://cathay-poc-os-admin.3scale.net/admin/api_docs/services/55457/preview
No Mapping Rule matched
I get this in the Nginx console on server startup
nginx: [warn] conflicting server name "ec2-107-22-18-40.compute-1.amazonaws.com" on 0.0.0.0:80, ignored
Same is this returns (remember it's a free heroku so first call every hour is slow):
http://two-methods-prints-headers.herokuapp.com/baggage/intl/items
Configs are unchanged - you can get them from 3scale.
I'm writing a module for Apicast v2. I used stapxx
to profile it, and realized that there was something wrong. Around 75-80% of the time was consumed on the method prequire()
of module.lua
.
My module defines rewrite()
, access()
, etc. instead of call()
. The problem is that in the load()
method of module.lua
there's this:
local files = {
[ name .. '.' .. phase ] = 'call', -- like apicast.init exposing .call
[ name ] = phase -- like apicast exposing .init
}
for file, method in pairs(files) do
local ok, ret = prequire(file)
and prequire()
only caches a result when it's successful. In other words, as I haven't defined call()
in my module, prequire()
fails, and it does not cache that it failed, so it retries in every single request. This is a huge performance hit. When I fixed this, I noticed that the time spent prequire()
went down from 75-80% to almost none.
With the new modular architecture, it'd be good to create some documentation which explains how to add custom behavior to the gateway by creating a module.
Using proxy_pass $var
will not use upstream and keep-alive connection.
Either we have to use cosockets and lua-resty-http to stream API request and responses, or use balancer_by_lua in upstream.
There is a number of parameters that can be set for APIcast that modify its behavior, normally they are set via environment variables (e.g. AUTO_UPDATE_INTERVAL
, APICAST_SERVICES
, APICAST_MISSING_CONFIGURATION
etc.)
All these parameters need to be documented: the name, what they do, what are the allowed values/format.
I think this needs to be a separate reference document.
We don't have the necessary test coverage of APIcast in OAuth mode (see #6593)
We should have integration tests with real IAM
We could consider something like https://github.com/doorkeeper-gem/doorkeeper
or http://www.ory.am/hydra
It seems that with latest releases we've lost the ability of using the user agent header to track source of calls to backend and we'd like to bring it back.
In addition, it'd be useful to also being able to track:
Currently when making the call to /admin/api/nginx/spec.json
the API can return: {"error":"Access denied"}
in case the access token is not correct, or {"status":"Not found"}
if the URL is wrong. This body is used to set provider.contents
.
download()
and curl()
in configuration.lua
should check for the status code of the response and if it's not 200
probably set the provider.contents
to null
.
Can the repo administrator enable squash merging on repo as per this article? Not sure I like how #150 is currently looking :)
I understand that with OAuth additions we have made that we have introduced a (optional) dependency on REDIS in apicast?
Can someone confirm/deny/explain it a bit more and the need.
I think it would be great it we could find other ways to solve the same need and keep the simple apicast deployment as simple as possible.
Luarocks is trouble. It needs to be installed with proper settings. Passed proper flags.
We can use soon to be released openresty package manager opm.
To add a custom plugin, we only need to define a lua module that implements a few methods like rewrite(), access(), etc. that are called in the appropriate nginx phases.
The problem is that as it is defined right now, the plugin system provided by Apicast forces custom plugins to define all those methods, even if they do not have anything to do with the functionality that the custom plugin adds. We should implement some changes so methods that are not likely to change from plugin to plugin do not need to be duplicated across all the custom plugins.
Mapping rules defined in 3scale API Manager can have additional query parameters which are taken into account for mapping:
For GET requests it's very straightforward โ the call:
curl -XGET "http://localhost:8080/test?key=hello&another=parameter"
will match the method get
as expected.
But for POST the behavior is a bit strange. The reason is that for "non-GET" requests the "query parameters" are taken from the body.
So, this request will not match:
curl -XPOST "http://localhost:8080/test?key=hello" -d "another=parameter"
while this one will match the pattern /test?key={value}
(post
method):
curl -XPOST "http://localhost:8080/test" -d "key=hello&another=parameter"
I am trying to understand if this is by design, or this behavior should be change.
I'd say the current behavior is not intuitive, and would only be partly useful if application/x-www-form-urlencoded
content type is used, as it will not work for JSON payloads for example.
I believe only actual query string needs to be taken into account for mapping rules matching.
Based in legal input, Apache2.0 is slightly better than MIT, and while all the contributions to the repo are 3scalers (and hence copyright is owned) we can easily switch the license.
So, I propose we do it now - before we accept any 3rd party contributions that complicate the switch.
APICAST_RELOAD_CONFIG can control reloading of the configuration, but it is not exposed as OpenShift parameter.
Not reloading.
Reloading config on every request.
[provide a description of the issue]
I am led to understand that OAuth is not working correctly in apicast at the moment.
Can anyone verify that?
Rename THREESCALE_ENDPOINT in OpenShift template to THREESCALE_ADMIN_URL as that better described what it is, is more consistent with other references we have - and we tend to reserve the "ENDPOINT" term for our APIs, not for a url of the UI.
Please do this asap and let me know, as the Tutorial text and screenshots all need updating.
Close to 100% of large complex on-boardings require some degree of customization of their Nginx configuration. Typically, changes are required around the server and location blocks inside Nginx.conf. Lua is less subject to change, though we frequently need to modify that as well. Anything from
-using URL Path to dictate which Service is used to
-adding custom headers to
-adding custom paths to IDP Servers to
-JWT validation etc.
Nginx Openshift needs a back door to slot in custom configuration.
This PR introduced routing the requests to different services based on the path, which allows having the same Public Base URL in multiple services.
This works, but the way it works is not very efficient. In the worst case it will iterate over all services and all mapping rules, and do regex matching for each one of them. After getting the service id, the results of regex matching are discarded, and it will be iterated again for all the rules of the service.
I think this can be improved.
As the 1st step it's probably worth doing a benchmark to understand better how serious the problem is.
Then, we can think about how this can be improved. I am thinking of two ways:
Use the find_service_cascade
function as it is now, but store the results of usage
and matched_patterns
to reuse them later and avoiding regex matching again.
On configuration load build some kind of index based on the mapping rules patterns (across all services), which will be used for service id lookup on incoming requests.
Currently the Management API can update the configuration with a JSON configuration file passed as body in POST /config
or PUT /config
.
It would be good to have a method that would invoke downloading the configuration from 3scale and applying it to the running gateway.
Make the THREESCALE_ADMIN_URL placeholder that appears in the UI (in the new app form from the template) consistent with the support site "standard" (i.e https://MYDOMAIN-admin.3scale.net/).
We are using quay.io/3scale/gateway:latest
at the moment, which is from 4 months ago. I'd like to use the v2
tag (seems to be quite a lot of development). Anything I should be worried about or do before upgrading? Thanks.
There are a number of questions that are asked frequently about apicast, we can help answer them proactively with a FAQ in the repo. I would put in docs folder and also link to it from the readme.
Deployment options
What OS is this project tested on (not implying support)
What version of OpenResty is recommended and tested against?
Failover. If the 3scale Service Management API is unreachable, does the API Gateway continue to work, how?
Let's rename this repository to APIcast.
apicast
quay.io/3scale/gateway
to quay.io/3scale/apicast
?@vramosp do you agree?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.