guinmoon / llmfarm_core.swift Goto Github PK

This is the error I am facing , I tried handling it with many if , guard but maybe its throwing error because of model is nil or its receiving nil ?

Not a real issue, but more a question.
current llama.cpp code supports loading also in lora adapters. Is this also supported by llmfarm_core.swift as I do not see that back in the LLMFarm code/application.
In case it is supported, is there an example on how to use it?

Hello

Each time I add llmfarm_core package as a dependency I see the following error:
The package product 'llmfarm_core' cannot be used as a dependency of this target because it uses unsafe build flags.

Any idea why?

Hey, I saw your repo, and I think that you're working on really cool stuff!

I was trying to hack out figuring out how to enable Metal on llama.cpp for Swift apps, and it looks like you were able to hack something together. Do you have any plans to bring your changes to llama.cpp, and if not, would you please share any insights into how you overcame them? I'm new to Swift, so I can't really make heads or tails of it but was able to run your repo w/ Metals successfully.

Hello!

The current implementation crashed when trying to load llama-2-7b.q4_K_M.gguf.

                .onAppear {
                    Task {
                        // Load AI model
                        let url = Bundle.main.url(forResource: "llama-2-7b.Q4_K_M", withExtension: "gguf")!
                        let scribe = AI(_modelPath: url.path(),_chatName: "chat")
                        var params: ModelContextParams = .default
                        params.use_metal = true
                        
                        do {
                            try scribe.loadModel(ModelInference.LLama_gguf, contextParams: params)
                        } catch {
                            // Handle the error
                            print("An error occurred: \(error)")
                        }
                    }
                }

at line 484 in ggml-metal: ctx->buffers[ctx->n_buffers].metal = [ctx->device newBufferWithBytesNoCopy:(void *) ((uint8_t *) data + i) length:size_step_aligned options:MTLResourceStorageModeShared deallocator:nil];

Thanks for this wrapper. Do you think it'd be possible to add support for the new Mistral model? It seems to be working according to https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF. Otherwise I'll submit a pull request by this weekend.

Hi there,

ReadMe mentions removing unsafe flags, but I am unable to do so if I add it as a Package Dependency.

Are you suggesting I have to manually build the source without the unsafe flags to use in my project?

Apologies if this sounds too simple of a question, I am figuring out how to build a Mac OS chat desktop application that interacts with a mistral GGUF model.

Thanks,
Chaks