Code Monkey home page Code Monkey logo

server-side-extension's Introduction

Server Side Extension

This repository provides a server-side extension (SSE) protocol based on gRPC that allows you to extend the Qlik built-in expression library with functionality from external calculation engines. You can use external calculation engines in both load scripts and charts. In Qlik Sense and QlikView, you connect to these server-side extensions (SSEs) by defining analytic connections.

This repository includes documentation that describes the SSE protocol, how to use it and how to build your own plugins. We have also examples written in a couple of different languages.

Status

CircleCI

Latest SSE Version: v1.1.0
Examples Disclaimer: The examples provided are just examples, therefore use them at your own risk.

Latest Product Version SSE Supported
Qlik Sense February 2018 v1.1.0
QlikView November 2017 v1.0.0

Previous Versions

Documentation

Examples

Python

C++

C#

Java

Go

Contributing

Please follow the instructions in CONTRIBUTING.md.

Getting help

  1. Look at the documentation and examples in this repo again.
  2. Do a search in Qlik Community to see if there is already an answer to your question.

If not, you can either

  • Ask your question in #sse on Qlik Branch slack. For access, go to Qlik Branch and qlik on SLACK. This is good for small questions and usually gives quick feedback.
  • Create a new question on Qlik Community. This is a good option for questions you think more people are interested in since question and answer will remain for others to see.

There are also several projects using SSE on Qlik Branch which can serve as useful examples.

server-side-extension's People

Contributors

bjornwa avatar ifreezer avatar jdmarlin avatar joesiltbergqlik avatar josefinestal avatar kjartan-qlik avatar linneastovring-nielsen avatar mjacobsson avatar peol avatar qlik-aoh avatar qlik-des avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

server-side-extension's Issues

Issues with Python in Docker container

Description

I do have some issues running server side extension in docker. The building of the docker files runs fine. per your instructions.
However, when running the image I get a Python error. I'm running docker in CentOS I have also tested this in Ubuntu - still receiving the same error.

Steps to Reproduce

  1. download the zip file and unzip it in correct directory
  2. build the image with:
    -docker build -t qlik-oss/sse-helloworld -f Dockerfile.helloworld .
    The build runs fine
  3. Run the image

sseissues
docker run --rm -p 50052:50052 qlik-oss/sse-helloworld

Actual behavior

Gets and error, see attached screenshot

Environment

Docker

Operating system

CentOS and Ubunut 16.04

Qlik Sense

[ ] Desktop

Versions

  • Qlik Sense November 2017 release

python: can't open file 'ColumnOperations': [Errno 2] No such file or directory

Description

#docker run --rm -p 50053:50053 qlik-oss/sse-columnoperations
python: can't open file 'ColumnOperations': [Errno 2] No such file or directory

Steps to Reproduce

  1. docker build -t qlik-oss/sse-columnoperations -f Dockerfile.columnoperations .
  2. docker run --rm -p 50053:50053 qlik-oss/sse-columnoperations
Expected behavior

The example should run

Actual behavior

python: can't open file 'ColumnOperations': [Errno 2] No such file or directory

Environment

Operating system
[ ] Windows
[X ] Linux


Dangerous function call

Hi,

I've found some little issues in your example code. At

header.ParseFromString(header_iterator->second.data());

and

header.ParseFromString(header_iterator->second.data());

In this case it would be more safe to use the ParseFromArray function because the data can contain zero bytes which will result in a parse error. This caused some problems in our sse because we are evaluating the metadata.

Kind regards,
Daniel John

Where do I store server certs?

Description

I have generated certificates for helloword example, and I see folder sse_helloword_server_certs have 3 files named root_cert.pem, sse_server_cert.pem, and sse_server_key.pem, but where do I put those files?

The instruction says this, but I don't understand what to do,
The SSE plugin is the server in the SSE communication, so copy and refer to the folder named sse_qliktest_server_certs in the plugin server.

Expected behavior

Secured connection is established when running the program

Actual behavior

Unsecured connection was established

Environment

Operating system
[X] Mac OS X
Qlik Sense
[X] Desktop

Functions with a variable number of parameters

Description

Currently the examples don't cater for functions that take in a variable number of parameters. After working on my Python SSE project I find this to be a common requirement:

  • A function that takes in optional parameters
  • A function that can be run with a variable number of measures
  • A function that requires multiple data series as input

At the moment I can only see one workaround, which is to concatenate all the required parameters/ measures into a string in Qlik and pass it as a single parameter. The SSE then has to parse the string and extract the parameters/ measures. While I have got this working, a better implementation would make the solution more robust.

Things I'd really like to be considered are:

  • A new DataType and Qlik expression syntax to pass a list/array of Duals as a parameter.
  • A better way to pass key word arguments to a function.

I'm happy to discuss this further and show example use cases.

Environment

Operating system
[x] Windows
[x] Linux
Qlik Sense
[x] Desktop
[x] Server

Versions

  • server-side-extension: v1.1.0
  • Qlik Sense: April 2018
  • Operating system: Windows Server 2012 R2, Windows 10
  • Python 3.6

SSE function call on demand from QlikView

Hey, we are currently experimenting with SSEs in QlikView and in Qlik Sense.

Now at the moment we got a little bit stuck...
What we want to do is triggering a function call on demand from our Extension to the SSE.
We found a (hacky) way to do this in Qlik Sense by defining a SSE function call within a Hypercube.
But we do not find a way to do so in QlikView.
Do you may have an idea how to solve this?

Changing the return types with TableDescription gives error in FullScriptSupport using Pandas

Description

Choosing a different data type of the returned columns in the tableDescription gives an error if the data type differs from the function's return type.

Steps to Reproduce

  1. Open the FullScriptSupport using Pandas app
  2. Navigate to the Data Load Editor
  3. In the vScript variable, change field.dataType = 1 to field.dataType = 0
  4. Reload the script.
  5. The reload will fail with the following error:
    image

Versions

  • server-side-extension: v1.1.0

Struggling to connect to Qlik

Description

Hi,
I have followed all the steps for the HelloWorld python example. I am using Qlik Sense Enterprise on the web.

When I run the main.py file I get a connection to the correct port, however when opening Qlik Sense and the .qvf app nothing is loaded, and the command prompt doesn't show the capabilities that should be listed.

I have setup certificates and they are in the right place and folder path has been set in the Analytical connection.

I cannot see what I am doing wrong and really need some help.

Steps to Reproduce

Certificates:
image

Analytical connection:
image

Folder with all scripts in for HelloWorld:
image

Command prompt when I run python main.py:
image

Result (or no result) to Qlik app:
image

Environment

Operating system
 Windows
Qlik Sense
[Server (web)

Versions

  • server-side-extension: One from this github repo
  • Qlik Sense: Most recent

Column example naming does not follow "Python - Gettings started"-guide

In the guide "Getting started with python" the example says to use e.g.
SSEPlugin=columnsupport,localhost:50053

Although when opening SSE_Column_Operations.qvf file the columnsupport is named Column and the app does not work as expected without changing this.

Expected
Homogeneous naming either columnsupport or Column in all places.

There are no Service Descriptors in the generated ServerSideExtension_pb2.py

Description

[Description of the issue]
Not able to correctly use reflection for the server.

Steps to Reproduce

  1. [Step one]
  2. [Step two]
  3. ...
Expected behavior

[What you expected to happen]
Trying to use reflection to gather information about the SSE server. The goal is to be able to use the health check service so we can monitor the server from afar.

Actual behavior

[What actually happened]
The generated ServerSideExtension_pb2.py does not have a service descriptors
Compare the generated ServerSideExtension_pb2.py with the helloworld_pb2.py from the grpc githubs' repo.
https://github.com/grpc/grpc/tree/v1.18.x/examples/python/helloworld

This brings up another question. If we add a service to the proto file which Qlik as a client does not need to know (say a check service), and generate a new ServerSideExtension_pb2.py, would that break the communication between Qlik and our server?

Environment

Operating system
[x ] Windows
[ ] Linux
Qlik Sense
[ ] Desktop
[ x] Server

Versions

  • server-side-extension:
  • Qlik Sense:
  • Operating system:
  • [Other relevant versions]

Python OS Command Execution via Examples

Description

All of the Python SSE servers currently support arbitrary command injection. Some of the other examples do a better job restricting functions to expected code input.

Mutual TLS should be in front of these servers, but in case it's not configured anyone can execute arbitrary OS commands fairly easily against the examples.

The eval and exec functions are unsafe for user input. It'd be good to update the examples with a more secure server for users to use as a template.

Steps to Reproduce

  1. Set up a Qlik analytic connection and run any of the example Python servers, such as helloworld
  2. Configure the extension editor in Qlik to run SSEPython.ScriptEvalStr('__import__("os").popen("ls -la").read()', HelloWorldData). Modify the output field if not using the SSE_Hello_World.qvf to another string field.
  3. Apply the expression and observe that the server executes ls -la on the host.

For columnoperations:
SSEPython.ScriptEval('__import__("os").popen("ls -la").read()', [], HelloWorldData)

For fullscriptsupport_pandas (which used exec instead of eval):
SSEPython.ScriptEval('__import__("os").popen("touch /tmp/x").read()')

The fullscriptsupport example can use the same payload as the helloworld example.

Expected behavior

The example servers should be secure against remote code execution vulnerabilities.

Actual behavior

Unexpected users can execute OS commands on servers hosting the Python examples.

Environment

Operating system
[ ] Windows
[X] Linux
Qlik Sense
[ ] Desktop
[X] Server

Versions

  • server-side-extension: Latest
  • Qlik Sense:
  • Operating system:
  • [Other relevant versions]

Message length limit

Description

I am hitting gRPC maximum message size limits when making a SSE call through the Qlik load script.

The following error occurred:
Connector reply error: grpc::StatusCode::RESOURCE_EXHAUSTED: 'Received message larger than max (5106076 vs. 4194304)'

I have already tried increasing the message length on the SSE server using the code below, but this does not seem to help. Is there any way to increase this limit?

MAX_MESSAGE_LENGTH = 50 * 1024 * 1024

server = grpc.server(futures.ThreadPoolExecutor(max_workers=10),\
        options=[('grpc.max_send_message_length', MAX_MESSAGE_LENGTH),('grpc.max_receive_message_length', MAX_MESSAGE_LENGTH)])

Steps to Reproduce

  1. Make an SSE call that returns a response > 4 MB.
Expected behavior

Ability to change message length limits.

Actual behavior

Changing message length limit for the SSE does not solve the issue.

Environment

Operating system
[x ] Windows
[ ] Linux
Qlik Sense
[ ] Desktop
[ x] Server

Versions

  • server-side-extension: Python SSE derived from Column Operations examples
  • Qlik Sense: June 2018
  • Operating system: Windows Server 2012 R2
  • Python 3.6

Error in visualization when selecting dimensions and check.

Description

For an SSE, I have a visualization and a list of values for my dimension to filter from. I select the dimension and select the green checkbox while the visualization is loading. This results in an error. I am currently running windows desktop version which I downloaded two days ago. The SSE is working fine connecting python to Qlik. The results are fine but without any fault of the extension, Qlik provides an error message stating "Error: Unonk error".

We were thinking to incorporate this in our firm but this bus makes all the great features to the side when the user experience will be errors one visualization.

Steps to Reproduce

  1. Install: https://github.com/nabeel-oz/qlik-py-tools
  2. Go to a sample app or create your own with python in the script.
  3. Make a visualization that runs Python
  4. Have a filter of a dimension of your choice.
  5. Select a value from the filter
  6. While the SSE is working with Python server to recompute values select the green checkbox to confirm your dimension selection.
  7. One shall be presented with the error message int eh visualization.
Expected behavior

The value is selected and the visualization loads.

Actual behavior

the value is selected and the visualization crashes to load

Environment

Enterprise or desktop.

Operating system
[X ] Windows
Qlik Sense
[X ] Desktop
[X ] Server

Versions

Everything is the latest. I even went as far as reinstalling windows in a machine to make sure everything was up to date.

The Analytics Connection between different machine with no certificates(SSE)

Hi,
I want to use server side extensions (SSE) for Qlik Sense and Python in a multi-node environment.
So I follow the installation steps 1 and 2, which refer to the URL below on the python server.
URL: https://developer.qlik.com/garden/5af5217ab2606a3c2c1f4d1d
Then I made Analytics Connection through QMC on the Qlik server.
Certificate file path information was not entered because a certificate file was not created.
Edit property HOST = IP address, port number only.
And the Qlik server and Python server communicate normally.
My question is,
Is communication possible without SSL installation and SSE certificate generation?

Additional Information
Using https://~ adrress.
Qlik Server Os = Windows Server 2016
Qlikver = Qlik Sense Enterprise 2019.2015
python server = Window11
python ver = 3.6.8

Does not create multiple connections to back end gRPC service

Description

I am working on building a gRPC Python container to provide specific functions to the Qlik Engine. The service works correctly and I get results back without problems. The challenge is that when I try to scale it out to large number of data sets (App Reload function calls) OR with large numbers of users (App Sheet/Visualization calls), although I have 10 load balanced containers running, only the first one that is attached to by the engine is used.

I am not sure if this is caused by the Python Connector, gRPC, or by the way that the Qlik Engine interacts with the gRPC Client. Can anyone let me know where I should start looking to help build out the scalability that is required for large scale applications?

Cannot open SSE_Column_Operations.qvf

Description

I am trying to follow the columnoperation example. But I cannot open the according qlik app provided (SSE_Column_Operations.qvf). When I start this app immediately get an error "unsupported file format".

Steps to Reproduce

  1. Store SSE_Column_Operations.qvf in the apps folder
  2. Open app SSE_Column_Operations
Expected behavior

App can be opened.

Actual behavior

App cannot be opened

Environment

Operating system
[x] Windows
[ ] Linux
Qlik Sense
[x] Desktop
[ ] Server

Versions

qliksensedesktop: 13.51.4

We are getting gRPC initialization failure error code 14

Description

[Description of the issue]

We are getting gRPC initialization failure error code 14 in the SSE log of engine (C:\programdata\qlik\sense\log\engine\trace) after applying the analytics connection without certificates (insecure).

Python is hosted on a different machine with version 3.11
helloworld SSE package is also running in insecure mode

It is working fine when we are running SSE plugin in localhost of QlikSense Server

Expected behavior

GetCapabilities should be printed as soon we hit apply the analytics connection

Actual behavior

gRPC initialization failed with error 14

Environment

Operating system
[ ] Windows : 2016 R2

##### Qlik Sense

[ ] Server - Qlik Sense Client Managed on windows with May 2023 IR

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.