Comments (15)
typpo
commented on June 12, 2024
I think the problem here is that you are trying to use perplexity and openai APIs at the same time. That is a drawback to the "just override OpenAI" approach to supporting Perplexity.
If you set apiKey for perplexity in the YAML file, rather than envar, I think that might temporarily workaround your problem:
- id: openai:chat:pplx-70b-chat-alpha
config:
apiHost: api.perplexity.ai
apiKey: xxxfrom promptfoo.
typpo
commented on June 12, 2024
#456 makes it possible to override apiKeys without having to hardcode them. Once it lands, you can do something like:
- id: openai:chat:pplx-70b-chat-alpha
config:
apiHost: api.perplexity.ai
apiKeyEnvar: PERPLEXITY_API_KEYThis makes it so you don't have to hardcode the API key in your config to get the desired result :)
from promptfoo.
typpo
commented on June 12, 2024
This change is published in 0.40.0
from promptfoo.
gillsgills
commented on June 12, 2024
@typpo So many thanks for the prompt reply and effort! And I apologize about my late response on this issue.
I switched the version of promptfoo to 0.40.0, and then added the suggested configuration into my YAML file:
providers:
- id: openai:chat:gpt-3.5-turbo-0613
- id: openai:chat:pplx-70b-chat-alpha
config:
apiHost: api.perplexity.ai
apiKeyEnvar: PERPLEXITY_API_KEY
But then, same error appeared. This time I actually noticed this message below at the beginning of terminal output:
Using unknown OpenAI chat model: pplx-70b-chat-alpha
Eval: [████████████████████████████████████████] 100% | ETA: 0s | 27/27 | openai:chat:pplx-70b-chat-alpha "Answer thi" question=W
But according to the perplexity tab in the documentation page, openai:chat:pplx-70b-chat-alpha shall be there right?
from promptfoo.
typpo
commented on June 12, 2024
The warning message is safe to ignore. I'll make a change to suppress it when apiHost is overridden.
from promptfoo.
gillsgills
commented on June 12, 2024
@typpo So the error should not be related to the problem of invalid model right? Which means openai:chat:pplx-70b-chat-alpha should be OK and there was something else went wrong.
from promptfoo.
typpo
commented on June 12, 2024
It's not actually an error. It's just a warning that pplx-70b-chat-alpha is not a known OpenAI model.
from promptfoo.
gillsgills
commented on June 12, 2024
@typpo Thanks for the fix on this issue. But the same error still exist unfortunately. I use below setup this time:
providers:
- id: openai:chat:gpt-3.5-turbo-0613
- id: openai:chat:llama-2-70b-chat
config:
apiHost: api.perplexity.ai
apiKeyEnvar: PERPLEXITY_API_KEY
And while the result from GPT-3.5 looks fine, the result from perplexity side is
[FAIL] API call error: Error: Error parsing response from https://api.perplexity.ai/v1/chat/completions: FetchError: invalid json response body at https://api.perplexity.ai/v1/chat/completions reason: Unexpected end of JSON input
from promptfoo.
typpo
commented on June 12, 2024
Thanks for following up @gillsgills.
#464 should solve this problem, as it looks like there is a minor incompatibility with the OpenAI API (lack of a version param in API route)
I've also tested it with perplexity API myself and added an example #463
from promptfoo.
gillsgills
commented on June 12, 2024
@typpo Thanks a lot for your prompt response and effort!
So since #464 is yet to be merged, I wonder if I can have a taste by installing from the local source? Or I may wait a bit for the sake of stability.
from promptfoo.
typpo
commented on June 12, 2024
Please give it a try in 0.41.0 :)
from promptfoo.
gillsgills
commented on June 12, 2024
Hi @typpo I know I have been really a nuisance here by keep having trouble on this question, but seems like it is still not working well.
Below is my local_path.sh setup. I source it to install the promptfoo:
npm install [email protected]
Then I run promptfoo eval -c test_output.yaml. test_output.yaml is like below
prompts:
- "Who discovered {{topic}}?"
providers:
# Compare GPT 3.5 vs Perplexity 70B
- id: openai:chat:gpt-3.5-turbo-0613
- id: openai:chat:pplx-70b-online
config:
apiBaseUrl: https://api.perplexity.ai
apiKeyEnvar: PERPLEXITY_API_KEY
frequency_penalty: 0.1
tests:
- vars:
topic: gravity
The output is
$ promptfoo eval -c test_output.yaml
Using unknown OpenAI chat model: pplx-70b-online
Eval: [████████████████████████████████████████] 100% | ETA: 0s | 2/2 | openai:chat:pplx-70b-online "Who discov" topic=grav
┌────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┐
│ topic │ [openai:chat:gpt-3.5-turbo-0613] Who discovered {{topic}}? │ [openai:chat:pplx-70b-online] Who discovered {{topic}}? │
├────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ gravity │ [PASS] Sir Isaac Newton is credited with discove │ [FAIL] API call error: Error: Error par │
│ │ ring gravity. │ sing response from https://api.perplexity.ai/v1/chat/compl │
│ │ │ etions: FetchError: invalid json response body at https:// │
│ │ │ api.perplexity.ai/v1/chat/completions reason: Unexpected e │
│ │ │ nd of JSON input │
│ │ │ --- │
└────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘
======================================================================================================================================================================================
✔ Evaluation complete.
Run promptfoo view to use the local web viewer
Run promptfoo share to create a shareable URL
Run promptfoo feedback to share feedback with the developers of this tool
======================================================================================================================================================================================
Successes: 1
Failures: 1
Token usage: Total 20, Prompt 11, Completion 9, Cached 0
Done.
I believe after all the struggles and problems, it can be either networking or my stupid mistake....what can it be?
from promptfoo.
typpo
commented on June 12, 2024
No problem, you are not a nuisance. Sorry this has been such a struggle to set up.
It's strange because I'm able to get that exact config working just fine. Can you double check that you've set the PERPLEXITY_API_KEY environment variable and that is in fact a valid API key? Alternatively, if you set apiKey instead of apiKeyEnvar, what do you see?
from promptfoo.
gillsgills
commented on June 12, 2024
Thanks a lot for understanding.
I have changed apiKeyEnvar to apiKey and same error occured. I tried my key on this website (right-hand side, "Bearer") and it can return proper answer. So I guess I have the right key? Or do you have any other right way to valid the key, if what I did is the wrong way to check?
{
"id": "5d3b56e8-7864-49b5-898d-3fa2cd7dc607",
"model": "mistral-7b-instruct",
"created": 9266019,
"usage": {
"prompt_tokens": 25,
"completion_tokens": 50,
"total_tokens": 75
},
"object": "chat.completion",
"choices": [
{
"index": 0,
"finish_reason": "stop",
"message": {
"role": "assistant",
"content": "The exact number of stars in the Milky Way galaxy is not known, but it is estimated to be between 100-400 billion. The latest estimate from astronomers suggests there could be around 200 billion stars."
},
"delta": {
"role": "assistant",
"content": ""
}
}
]
}
from promptfoo.
gillsgills
commented on June 12, 2024
Hi @typpo sorry from my side for being busy with other issues. Any follow-up on this?
from promptfoo.
Related Issues (20)
- CLI docs request: default behavior of `eval --output` HOT 1
- Support `systemInstruction` for Gemini (PALM) HOT 4
- Allow options to avoid using special characters HOT 4
- How can I set threshold for avg. of test cases scores with csv? HOT 2
- Types for `promptfoo.evaluate` broken when using TS
- Add ability to bust cache from the web UI
- When Python provider raises exception, details are no longer recorded HOT 3
- Ensure percent complete accurately reflects test suite
- Migrations path incorrect in self host docker build HOT 4
- [Web UI] Image previewer not working for variables or failed tests
- CI passes despite failing build HOT 1
- python external assertion not working HOT 2
- [Web UI] Increase robustness of markdown rendering
- Specify a label with a prompt function HOT 3
- --no confirmation for cli view HOT 1
- expression to select subset of output to display in view HOT 3
- SqliteError: no such table: evals HOT 1
- Error: Unknown Amazon Bedrock model: meta.llama3-70b-instruct-v1:0 HOT 4
- Compare providers based on latency HOT 3
- Failed to fetch when attempting to log in/sign up on the web-UI after promptfoo share
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from promptfoo.