Comments (8)
Hi @doggy8088
Thanks for the suggestion. Yes, I was thinking about a retry mechanism already and considering something like Polly. However, right now I'd like to keep dependencies on other packages as small as possible and this package supports .NET Framework 4.7.2, .NET Standard 2.0, and .NET 6 or higher.
Do you have suggestions on how to implement a "native" retry mechanism?
from generative-ai.
I think a built-in caching mechanism would be more convenient. It could be a optional argument to enable/disable this feature. Or provide a standalone method to do that. It could be a helper method or static method or extension method.
For the retry mechanism, this should be abstract to the developer. The method only throw exception when it reached max retry limit.
from generative-ai.
Here is my code snippet for retry mechanism:
public async Task<string> Run(double temperature = -1, int retries = 5, string[] stopSequences = null)
{
string endpoint = $"https://generativelanguage.googleapis.com/v1beta/models/gemini-1.0-pro-001:generateContent?key={this.ApiKey}";
var request = new GeminiRequest()
{
Contents = this.Contents.ToArray(),
GenerationConfig = this.DefaultGenerationConfig,
SafetySettings = this.DefaultSafetySettings
};
if (temperature >= 0)
{
request.GenerationConfig.Temperature = temperature;
}
if (stopSequences is not null)
{
request.GenerationConfig.StopSequences = stopSequences.ToList();
}
var json = request.ToJson();
using (HttpClient client = new HttpClient())
{
var content = new StringContent(json, Encoding.UTF8, "application/json");
var message = await client.PostAsync(endpoint, content);
responseContent = await message.Content.ReadAsStringAsync();
this.LogResponseJson(responseContent, logId: logId);
if (!message.IsSuccessStatusCode)
{
if (retries <= 0)
{
throw new Exception($"Gemini Error: {message.StatusCode}\n\n{message.Content.ReadAsStringAsync().Result}");
}
Thread.Sleep(5000);
return await Run(temperature, retries - 1, stopSequences);
}
}
GeminiResponse response = GeminiResponse.FromJson(responseContent);
var firstCandidate = response.Candidates.First();
bool status = false;
switch (firstCandidate.FinishReason)
{
case FinishReason.SAFETY:
// Util.HorizontalRun(true, request, response).Dump();
break;
case FinishReason.MAX_TOKENS:
// Util.HorizontalRun(true, request, response).Dump();
break;
case FinishReason.RECITATION:
// Util.HorizontalRun(true, request, response).Dump();
break;
case FinishReason.OTHER:
// If you see BlockedReason.OTHER, the query or response may violate the terms of service or be otherwise unsupported.
// https://ai.google.dev/docs/troubleshooting#safety_issues
// Util.HorizontalRun(true, request, response).Dump();
break;
case FinishReason.UNSPECIFIED:
// Util.HorizontalRun(true, request, response).Dump();
break;
case FinishReason.STOP:
status = true;
break;
}
if (!status)
{
if (retries <= 0)
{
throw new FinishReasonException($"Error: {firstCandidate.FinishReason}");
}
Thread.Sleep(5000);
return await Run(temperature, retries - 1);
}
return firstCandidate.Content.Parts.First().Text;
}
from generative-ai.
I agreed with you about the HTTP 400 and 404. But for HTTP 500, that definitely need implement retry with backoff mechanism.
from generative-ai.
I'm going to look into that and open a feature branch.
The retry pattern for asynchronous calls in C# seems to be an interesting approach to solve this.
In case that you have code snippets or solid code samples, please feel free to post. That would be very much appreciated.
from generative-ai.
Dear @doggy8088
Thanks for the code snippet. I'm trying to get my head around several things here.
- When shall a retry be conducted? Depending on the returned HTTP status code there are not many options available or making sense to attempt a retry. The main reasons would be an
HttpIoException
or timeout related issues. As for any HTTP 4xx or HTTP 5xx I don't quite see the justification for a retry though. Perhaps an HTTP 429 could be of interest.
Do you see other reasons? - The handling of FinishReason seems to be contrary to the intended purpose (while reading the Python sources). However, I agree that the current implementation needs improvement in that area.
- Your approach seems to work for non-streaming requests only. Or am I missing something?
As mentioned before, I like the idea to add the retry design pattern to the package. I'm currently a bit undecided about the actual how and the implementation.
Again, thanks for your contribution.
PS: The way how HttpClient
is used can cause trouble regarding port exhaustion on the client. There are safer ways.
from generative-ai.
In my personal experience, almost every circumstance needs retry. There are so many false positives error returned from Google AI Gemini API. The FinishReason could lie. In my current implementation, I retry everything when I get non-200 response.
I never tried streaming till now.
from generative-ai.
I see.
In case of an HTTP 400 Bad Request or an HTTP 404 Not Found, I wouldn't go for a retry. Same in case of an HTTP 500 Server error or similar.
I get your reasoning, and I'm giving it a bit more thought. In general, I like the retry design pattern, and yes I'd like to implement it as well into the package. Bear with me a little.
from generative-ai.
Related Issues (20)
- Discussion: Keep API interface consistent HOT 3
- Support: Gemini Pro Vision HOT 7
- The original usage seems broken: new GenerativeModel(...) HOT 4
- Feature suggestion: Add logs with LogLevel using the Standard logging in .NET HOT 11
- Unknown error due to some settings HOT 5
- SafetySettings can be easier and less error-prone. HOT 5
- Application Default Credentials (ADC) has been loaded automatically even I use API Key auth. HOT 4
- The `_useHeaderApiKey` is always be `true` when using API Key auth. HOT 2
- Hints for possible Exceptions HOT 5
- Check for more FinishReason HOT 4
- `model.GetModel()` returned wrong `SupportedGenerationMethods` HOT 2
- A bug is found on `EmbedContentRequest` HOT 1
- Provide full context for the HttpRequestException HOT 6
- Assigning a API_KEY using model.ApiKey is not working HOT 3
- The `inheritdoc` XML comment can't appear in LINQPad HOT 1
- await model.ListFiles() get HTTP 403 (Forbidden) HOT 4
- Feature suggestion: create a Tool object from a C# method HOT 1
- Error while copying content to a stream HOT 2
- Exception thrown in Google App Engine HOT 3
- Correct usage of systemInstruction
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from generative-ai.