Comments (7)
Hey @DhruvThu, can you share a few id
s from the responses you get back on vertex requests? Or share a few request ids? This will help us debug.
from anthropic-sdk-python.
Thanks for responding. Also, I am using streaming response from vertex anthropic and these are some of the message ids which i got in first chunk. msg_01D2jNpu4rUZMXUvwtpipMnx, msg_01CMjRdPAhDQaWELbrgSirS8
from anthropic-sdk-python.
Hmm, message ids from vertex should look like msg_vrtx_...
. The ids you shared are for the direct (1P) Anthropic API.
from anthropic-sdk-python.
Could you check for this? msg_vrtx_01AaDL52fwpTrqFftLMxxQ1e. Sorry for the previous one. In this message, it took around 2.4s to start streaming.
The streaming response through direct Anthropic API took around 0.89s. The message id for that is msg_01AWgnspZ2w5NhzE92uL7VZ9.
The code I am using is as follows,
class AnthropicLLM:
def __init__(self, anthropic_client : Anthropic, anthropic_vertex_client : AnthropicVertex) -> None:
self.anthropic = anthropic_client
credentials, project_id = google.auth.load_credentials_from_dict(google_credentials_info, scopes=["https://www.googleapis.com/auth/cloud-platform"],)
anthropic_vertex_client._credentials = credentials
self.vertex_anthropic = anthropic_vertex_client
self.messages = Messages(self)
class Messages:
def __init__(self, client : AnthropicLLM) -> None:
self.client = client
def create(self, model : str, messages : list[Message], temperature : float, system : str, stream : bool, max_tokens : int, tool_choice : str, tools : list[dict]):
model = model.replace("@", "-")
import time
st = time.time()
if(tools == []):
response = self.client.vertex_anthropic.messages.create(
model=model,
messages=messages,
temperature=temperature,
system=system,
stream=True,
max_tokens=max_tokens
)
print(response)
print(time.time()-st)
return response
else:
return self.client.vertex_anthropic.messages.create(
model=model,
messages=messages,
temperature=temperature,
system=system,
stream=stream,
max_tokens=max_tokens,
tool_choice=tool_choice,
tools=tools
)
from anthropic-sdk-python.
Hey @DhruvThu, we've identified the root cause of this issue. While we work on a fix you can workaround this issue by explicitly passing an access_token
, e.g. AnthropicVertex(access_token=access_token)
from anthropic-sdk-python.
Hey, thanks for the response. I try with access token
from anthropic-sdk-python.
This will be fixed in the next release, v0.30.2
, #573.
Note that you will still see a delay when making the very first request with an AnthropicVertex
instance as we need to fetch the access token but subsequent requests will use the cached token.
from anthropic-sdk-python.
Related Issues (20)
- Which Pydantic class to use for streaming messages? HOT 1
- Support String In ToolResultBlock param? 🧌 HOT 3
- Support Tool Choice Type: None
- httpx.PoolTimeout, fixed by disabling HTTP keepalive HOT 5
- Support Parallel Tool Calls = False
- Feature-request: Pydantic validators+serializers to be able to round-trip all supported types HOT 11
- API Requests Within Existing Projects HOT 2
- AnthropicVertex does not expose `with_options` HOT 3
- Error in the vertex AI documentation HOT 2
- Missing token count feature HOT 1
- Vertex "Could not resolve API token from the environment" HOT 9
- Wrong request options in retries HOT 4
- Is there a way to manage Claude Projects in Web UI via the API? HOT 1
- nest_asyncio hangs AsyncAnthropic
- `timeout` parameter doesn't seem to have any effect HOT 1
- Feature request: return the number of failed attempts HOT 2
- Error Code 400 HOT 2
- Async usage is broken as of 0.29.0 for client.messages.create HOT 3
- Feature request: tool_choice='none' for analysing available tools - but not using them HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from anthropic-sdk-python.