https://app.gitbook.com/s/EUbtdlfiGuPDocsqj2ZH/reference/

Vaex First

Typo: "Vaex is an open-soruce..."
Fix: "Vaex is an open-source"

Typo: "...to allow the extreme flexibility for advance pipeline solutions..."
Fix: "...to allow the extreme flexibility for advanced pipeline solutions..."

Best Practices

Add columns!

Typo: "...for every value tou would want..."
Fix: "...for every value you would want..."

Typo: "...which explain the XGBoost prediction."
Fix: "...which explains the XGBoost prediction."

Typo: "..., prediciton with distance (for confidance) etc,."
Fix: "..., prediction with distance (for confidence) etc,."

DataFrames

Typo: "In production, this allow you do make sure you... and passthrough elements you..."
Fix: "In production, this allows you to make sure you... and pass through elements you..."

Big Data -> Vaex

Typo: "Vaex is excellent for big data - lazy evlaution...
Fix: "Vaex is excellent for big data -- lazy evaluation..."

Variables and description

Typo: "... - any constant you what the backend/frontend could query."
Fix: "... - any constant you want, the backend/frontend could query." -> assuming this is the sentence you were going for

Advance -> Fix: "Advanced"

Complicated pipelines

sklearn_vs_vaex_vs_pyspark.ipynb

Typo: "..., you should give her a rise!"
Fix: "..., you should give her a raise!"

Ensembles with LightGBM, XGBoost, and CatBoost

ensemble_example.ipynb

Typo: "Crazy ensmble logic example"
Fix: "Crazy ensemble logic example"

Data science examples

Vaex Skleran Predictor -> Vaex Sklearn Prediction

Typo: "The predictor can apply any skleran..."
Fix: "The predictor can applly any sklearn..."

LightGBM

lightgbm.ipynb

Typo: "Variebels and description"
Fix: "Variables and description"

Typo: "...which want to assosiate..."
Fix: "...which want to associate..."

Typo: "A greate place..."
Fix: "A great place..."

Add the bytes 'Goldilox' at the beginning of the model bytes after save.
This will allow validating the model based on the first few bytes.

• Train
• Stream Inference
• Batch inference
• Serverless inference
  References:
• BYO containers

from_sklearn(..., inference_steps=None)

This will allow a pipeline for training, but removing some steps for inference
The main use for it is cleaning data in re-fit.

pipeline = Pipeline.from_sklearn(pipeline, inference_steps=[1,2, 5])

pipeline.fit(X, y) # uses all steps
inference(X, y) # uses only steps [1,2,5]

Implement a general MLFlowPipeline and from_mlflow.

• Much work.
• Good for completeness.
• Might be very complicated.

in gl serve <path> param1=param2.
Make sure all parameters move correctly with gunicorn.

Implement 'export_skops'
reference

https://trstringer.com/read-write-azure-storage-blob-python/

• gcpfs

When we run form_vaex(), or from_sklearn().
Run self.validate() before returning.

Polars lazy dataframe can be serialised.

Pseudo idea

1. We need to serialize a lazy frame into a "state"
2. Remove "input" and replace with some Special token.
3. Load new data, take it's input, insert to the "state".

• Unclear how to find the exact location as it can be adjusted.
• Might deal with selections - remove by default or keep.

5. result = pl.LazyFrame.read_json(io.BytesIO(json.dumps(state).encode())).lazy()

It could work in theory.

Implement a OnnxPipeline with from_onnx.

• Might be a lot of work for no value.
• Great for completeness
• Allow pyspark pipelines.

Add the branch and repo automatically.

reference

Add a glx build wich build a docker image for you

glx build <pipeline path> <image_name>

Should apply this for every printing in the project.

• Readme
• readthedocs
• Notebook examples
  • LightGBM
  • XGBoost
  • CatBoost
  • SKlearn
  • Advance LightGBM
  • Advance Sklearn
  • Keras (deep learning)
  • Tensorflow (deep learning)
  • Pytorch (deep learning)
  • Mxnet (deep learning)
  • nmslib (Nearest neighbors)
  • hnswlib (Nearest neighbors)
  • Faiss (Nearest neighbors)
  • sklearn nearest neighbors (Nearest neighbors)
  • implicit (recommender)
  • Intercept (general)
  • Flaml (automl)
  • auto-sklearn (automl)
  • auto-keras (automl, deep learning)
  • River (online leraning)

Add a description attribute