iusztinpaul / hands-on-llms Goto Github PK

🦖 𝗟𝗲𝗮𝗿𝗻 about 𝗟𝗟𝗠𝘀, 𝗟𝗟𝗠𝗢𝗽𝘀, and 𝘃𝗲𝗰𝘁𝗼𝗿 𝗗𝗕𝘀 for free by designing, training, and deploying a real-time financial advisor LLM system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 𝘷𝘪𝘥𝘦𝘰 & 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

License: MIT License

Python 7.60% Makefile 0.40% Shell 0.47% Dockerfile 0.03% Jupyter Notebook 91.50%

bytewax comet-ml huggingface mlops qdrant transformers beam langchain generative-ai llms

hands-on-llms's Introduction

🤖 Paul Iusztin

Senior Machine Learning Engineer • MLOps • Contractor ~ Helping startups engineer production-ready ML/AI systems.

About Me

I am a senior machine learning engineer and contractor with 5+ years of experience. I design and implement modular, scalable, and production-ready machine learning systems for startups worldwide.

🔥 My true passion is machine learning engineering.

💛 Secretly in love with software engineering.

🎨 I enjoy sharing my knowledge through creating content about designing and productionizing ML systems.

🪙 I am a blockchain and investing enthusiast.

👱🏻 Because I am not 100% a robot, I am also excited about self-development, psychology, cooking, hiking, skiing and my favorites 🐈‍⬛ cats.

👉 Check out The Full Stack 7-Steps MLOps Framework hands-on free course, where you will learn how to design, train, serve, and monitor an end-to-end ML batch system ~ source code + 2.5 hours of reading & video materials.

About my primary skills

⚫️ My top technologies ⚫️	⚫️ My top interests ⚫️
• Python, SQL • PyTorch, Scikit-Learn • NumPy, Pandas • AWS, GCP • Docker, FastAPI, Airflow, Kafka, Spark • DVC, Weights & Biases, MLFlow • PostgreSQL, Elasticsearch, Redis	• MLOps • generative AI • computer vision • recommender systems • batch & online serving • continuous training & monitoring • REST API, gRPC & streaming design • cloud & microservices • distributed systems

💬 Do you need machine learning solutions for your business? Let's discuss!

Let's connect ↓

↳ Subscribe to my weekly newsletter: Decoding ML

🎨 Creating content takes me a lot of time. If you enjoyed my work, you could support me by:

joining Medium through my referral link, you can support me without any extra cost while enjoying limitless access to Medium’s rich collection of stories;
buying me a coffee.

Thank you ✌🏼 !

hands-on-llms's People

Contributors

Stargazers

Watchers

Forkers

douglasmendes plushpluto mindkhichdi eliekawerk raghavsrivastava25 mhatrep rishabh135 zanche danieladama techthiyanes nehaa28 damonclifford skaiphd mekongdelta-mind danielmicoski drasaadmoosa mohnishbasha timadeg analysent omkar-sh aweshpd riverron polya20 tharwaniajay antonblockchain ramarnat trinpu977 solab5 atsyplikhin munirabobaker suganc alialemimatinpour moncefka hydrogeohc abhinavsmart clever-boy rizwandel jeremyengram tolulade-a hurricanejin majiouk dantegpt bharatr21 tuhinmallick sbusso thanhpham1987 mvandermeulen rajendra-ai biyanisuraj inde5media dipanjan001 taltaf913 dattgoswami sgatea ego pablosls sangemu735 si3mshady skp80 rmahidhar usct01 tiro2000 jokergithu aesthethic0de markusdegen sarkararnab91 jocirugeda jordilucas16 bluekiji77 sergiorgiraldo maaz-amjad hariprasad-1999 mariumaz gordonwho amik-aa artificialcoder02 cinderzhang aks861999 mehmetcanbudak rezalahmi anup77jindal kojokesse stevenxchung learnersj sumantabasak lgs lpai-org kevintsai wsanjay krishnamanoj-kota ramnathv tnt305 tienbku minhhoang270695 hoangfvx dauvannam321 vinayak-shanawad jonasbechthold jimmywang-jw hanzla-nawaz

hands-on-llms's Issues

TypeError: 'NoneType' object is not subscriptable

`Traceback` (most recent call last):
  File "/mnt/cephfs/home/shixun2024/miniconda3/envs/GengN/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/mnt/cephfs/home/shixun2024/miniconda3/envs/GengN/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/cephfs/home/shixun2024/users/GengNan/hands-on-llms/modules/training_pipeline/tools/train_run.py", line 83, in <module>
    fire.Fire(train)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/beam/app.py", line 1346, in wrapper
    return func(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/users/GengNan/hands-on-llms/modules/training_pipeline/tools/train_run.py", line 79, in train
    training_api.train()
  File "/mnt/cephfs/home/shixun2024/users/GengNan/hands-on-llms/modules/training_pipeline/training_pipeline/api/training.py", line 228, in train
    trainer.train()
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2654, in training_step
    loss = self.compute_loss(model, inputs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in compute_loss
    outputs = model(**inputs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
    output.reraise()
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
TypeError: Caught TypeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/peft/peft_model.py", line 922, in forward
    return self.base_model(
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 900, in forward
    transformer_outputs = self.transformer(
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 789, in forward
    outputs = torch.utils.checkpoint.checkpoint(
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 107, in forward
    outputs = run_function(*args)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 785, in custom_forward
    return module(*inputs, use_cache=use_cache, output_attentions=output_attentions)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 453, in forward
    attn_outputs = self.self_attention(
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 307, in forward
    query_layer, key_layer = self.maybe_rotary(query_layer, key_layer, past_kv_length)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 107, in forward
    cos, sin = self.cos_sin(seq_len, past_key_values_length, query.device, query.dtype)
  File "/mnt/cephfs/home/shixun2024/.cache/pypoetry/virtualenvs/training-pipeline-0MYnI07r-py3.10/lib/python3.10/site-packages/transformers/models/falcon/modeling_falcon.py", line 101, in cos_sin
    self.cos_cached[:, past_key_values_length : seq_len + past_key_values_length],
TypeError: 'NoneType' object is not subscriptable

I always meet this problem,how can i solve it?

Unable to find installation candidates for torch (2.0.1+cpu)

Great course so far!

I'm having some trouble running make install in the streaming_pipeline folder. Any suggestions?

Main dependencies are all installed.

The Trained model is not providing any response

I trained model end to end and once i deployed it on gradio for chat assistant, It was giving empty reponse for all the query

Exception: none is not an allowed value (type=type_error.none.not_allowed)

When I executed "make run_real_time" in folder "hands-on-llms/modules/streaming_pipeline", following instruction, an exception came up.

RUST_BACKTRACE=full poetry run python -m bytewax.run tools.run_real_time:build_flow
2024-03-12 10:14:28,492 - INFO - Initializing env vars...
2024-03-12 10:14:28,492 - INFO - Loading environment variables from: .env
2024-03-12 10:14:31,429 - INFO - HTTP Request: GET https://e4bf1959-f74c-490c-9f46-00897295ec9e.us-east4-0.gcp.cloud.qdrant.io:6333/collections/alpaca_financial_news "HTTP/2 200 OK"
Traceback (most recent call last):
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 88, in send
    return parse_as_type(response.json(), type_)
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 201, in parse_as_type
    return model_type(obj=obj).obj
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[InlineResponse2005] (for parse_as_type)
obj -> result -> config -> optimizer_config -> max_optimization_threads
  none is not an allowed value (type=type_error.none.not_allowed)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/bytewax/run.py", line 430, in <module>
    kwargs["flow"] = _locate_dataflow(module_str, attrs_str)
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/bytewax/run.py", line 204, in _locate_dataflow
    dataflow = attr(*args, **kwargs)
  File "/media/pactera/shared/Ubuntu/handsonllms/hands-on-llms/modules/streaming_pipeline/tools/run_real_time.py", line 26, in build_flow
    flow = flow_builder(model_cache_dir=model_cache_dir, debug=debug)
  File "/media/pactera/shared/Ubuntu/handsonllms/hands-on-llms/modules/streaming_pipeline/streaming_pipeline/flow.py", line 57, in build
    flow.output("output", _build_output(model, in_memory=debug))
  File "/media/pactera/shared/Ubuntu/handsonllms/hands-on-llms/modules/streaming_pipeline/streaming_pipeline/flow.py", line 90, in _build_output
    return QdrantVectorOutput(
  File "/media/pactera/shared/Ubuntu/handsonllms/hands-on-llms/modules/streaming_pipeline/streaming_pipeline/qdrant.py", line 42, in __init__
    self.client.get_collection(collection_name=self._collection_name)
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/qdrant_client.py", line 844, in get_collection
    return self._client.get_collection(collection_name=collection_name, **kwargs)
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/qdrant_remote.py", line 1566, in get_collection
    result: Optional[types.CollectionInfo] = self.http.collections_api.get_collection(
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/http/api/collections_api.py", line 838, in get_collection
    return self._build_for_get_collection(
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/http/api/collections_api.py", line 336, in _build_for_get_collection
    return self.api_client.request(
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 68, in request
    return self.send(request, type_)
  File "/media/pactera/shared/Ubuntu/python_poetry/cache/virtualenvs/streaming-pipeline-q1Qy2sa1-py3.10/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 90, in send
    raise ResponseHandlingException(e)
qdrant_client.http.exceptions.ResponseHandlingException: 1 validation error for ParsingModel[InlineResponse2005] (for parse_as_type)
obj -> result -> config -> optimizer_config -> max_optimization_threads
  none is not an allowed value (type=type_error.none.not_allowed)
make: *** [Makefile:31: run_real_time] Error 1

Why was this happening and what should I do? Thank you.

Missing train_finqa.py file required for dev_train_beam target in Makefile

Description

I am trying to execute the dev_train_beam target from the Makefile, but it seems like the train_finqa.py file, which is supposed to be located in the ./tools directory, is missing from the repository.

Steps to Reproduce

Navigate to the training_pipeline directory.
Run the command make dev_train_beam.

Expected Behavior

The make dev_train_beam command should execute successfully by running the Beam training pipeline using the train_finqa.py script.

Actual Behavior

An error is thrown indicating that the train_finqa.py file is not found:

Additional Information

I have searched through the repository to find the train_finqa.py file but was unable to locate it.
There is no documentation explaining the absence of the train_finqa.py file or how to generate or locate it.

Questions

Should the train_finqa.py file be included in the repository?
Are there any steps or commands that need to be executed to generate or locate the train_finqa.py file?
Is there any additional documentation available that explains how to set up and execute the dev_train_beam target?

Thank you for your assistance!

Running on aws ubuntu

RUST_BACKTRACE=full poetry run python -m bytewax.run tools.run_real_time:build_flow
2024-03-16 01:16:27,321 - INFO - Initializing env vars...
2024-03-16 01:16:27,322 - INFO - Loading environment variables from: .env
2024-03-16 01:16:30,824 - INFO - HTTP Request: GET https://475ea256-3440-4488-8d03-627b9c0f6ba0.us-east4-0.gcp.cloud.qdrant.io:6333/collections/alpaca_financial_news "HTTP/2 200 OK"
Traceback (most recent call last):
File "/home/ubuntu/.cache/pypoetry/virtualenvs/streaming-pipeline-En05NEtH-py3.10/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 88, in send
return parse_as_type(response.json(), type_)
File "/home/ubuntu/.cache/pypoetry/virtualenvs/streaming-pipeline-En05NEtH-py3.10/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 201, in parse_as_type
return model_type(obj=obj).obj
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[InlineResponse2005] (for parse_as_type)
obj -> result -> config -> optimizer_config -> max_optimization_threads
none is not an allowed value (type=type_error.none.not_allowed)

CM and CT

In your pipeline design, how do you implement Continuous Monitoring and Continuous Training, to account for language model drift over time?

Using fine-tuned model for inference

Hi @iusztinpaul,

Love the course so far!

I have a question: Shouldn't us use our fine-tuned model for inferenced instead of using Paul's peft model here?

hands-on-llms/modules/training_pipeline/configs/dev_inference_config.yaml

Line 2 in 5c1887b

id: iusztinpaul/fin-falcon-7b-lora:1.0.5

If yes then how should we publish our model from experiment to Comet Model Registry? Is it done manually via the Register Model button in the Comet experiment console view?

Thanks!

Unable to build container: rpc error: code = Unavailable

I'm getting the below error while running make dev_train_beam. I couldn't able to figure it out. I'm working on Mac OS. Your help is much appreciated