can we add a new signal option? it will be more flexible and can be used to abort the stream when a user cancels chat sessions. it meets the timeout requirement as well, timeout is also very useful as it's more convenient BTW :)

const model = genAI.getGenerativeModel({ model: "gemini-pro" }, {
  signal: AbortSignal.timeout(10000),
});

https://github.com/google/generative-ai-js/blob/d8f22d2259c9e926dcc4ac955db896c235e0a3bc/packages/main/src/requests/request.ts#L103-L117

Originally posted by @tmkx in #31 (comment)

const BASE_URL = "https://generativelanguage.googleapis.com";

The BASE_URL is not configurable.

It'd be helpful to make it so for passthrough proxies as needed. Also useful for testing and mocking out with local listeners if needed.

I scoured the docs for an example of how to specify the apiVersion parameter in a generate content request. I couldn't find it so I did a hard trace on the fetch() and finally figured out that it has to be in the call to getGenerativeModel():

    const model =
      getGenerativeModel(
        { model: modelName },
        {
            apiVersion: 'v1beta',
            configuration: generationConfig
          },
        );

I believe this is about to trip up a lot of people up due to the huge number of participants in the Google AI Hackathon going on right now. The contest wants you to use the Gemini 1.5 Pro advanced API and that is only available with the v1beta API. So initially my API calls to that model were failing until I figured out how to declare request option that properly.

Post-mortem I did manage to find an example of it buried in the node /advanced-function-calling.js sample. Given it's importance I feel it should be added to the the man repo README page and also at least somewhere on the main API docs page for JavaScript developers:

https://ai.google.dev/tutorials/get_started_web

If you scan that page, you will find many getGenerativeModel() call examples, some even showing how to declare SAFETY settings, but none of them show you where to put the apiVersion option.

Expected Behavior

I tried to create a simple Telegram chatbot for personal use, using cloudflare workers. All their IP Ranges are from an Available Region, and so is my acount where I generated the API Key. My use case falls strictly within the Terms

Actual Behavior

Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent: [400 Bad Request] User location is not supported for the API use.

Steps to Reproduce the Problem

1. Run this code in a (deployed) cloudflare worker (:

const genAI = new GoogleGenerativeAI(GEMINI_API_KEY);
const model = genAI.getGenerativeModel({ model: "gemini-pro"});
const prompt = "Write a story about a magic backpack."

const result = await model.generateContent(prompt);
const response = await result.response;
const text = response.text();

It always returns a 400 Bad Request error.

Firstly, Thanks for the work on this library :)

Expected Behavior

When an error occurs while using the API, the error message should be structured in a way that makes it easy to extract the reason for the failure. Currently, the error message is all squashed into one, making it tedious to get the reason property from the object, as shown in the following image:

Additionally, since the error is of the type any, it makes it difficult to tap into the reason for the failure, as shown in the following image:

Workaround

To work around this issue, the user has to manually parse the error message to extract the reason for the failure, as shown in the following code snippet:

const makeTheCall = async (
    message: object,
    apikey: string,
    generationConfig: object,
) => {
    await run(message, apikey as string, generationConfig)
        .then((response) => {
            settledState(response as string);
        })
        .catch((error) => {
            if (error.message.includes('[GoogleGenerativeAI Error]')) {
                const errorMessage = error.message;

                if (errorMessage.includes('[400 Bad Request]')) {
                    // Extract the reason for the error
                    const reasonStart = errorMessage.indexOf('reason":"') + 9;
                    const reasonEnd = errorMessage.indexOf('","domain');
                    const reason = errorMessage.substring(
                        reasonStart,
                        reasonEnd,
                    );

                    console.log('Error reason:', reason);
                }
            }
            // errorState(error.reason as string);
            throw new Error(error.message as string);
        });
};

This workaround allows the user to extract the reason for the failure, as shown in the following image:

Proposed Solution

To improve the error handling and provide more detailed error messages, I suggest the following:

1. Implement a custom error class that extends the built-in Error class and includes properties for the error code, reason, and any other relevant information.
2. When an error occurs, create an instance of the custom error class and throw it instead of the raw error object.
3. Update the error handling logic in the client code to handle the custom error class, making it easier to extract the relevant information from the error.

This approach will provide a more structured and informative error handling experience for the users of the library.

Repo Link

The repository where I encountered this issue is Genie.

Expected Behavior

model gemini-pro-vision generateContentStream with Chinese prompt returns malfunctioning chunks

Actual Behavior

model gemini-pro-vision generateContentStream with Chinese prompt returns malfunctioning chunks which contains ���.

Steps to Reproduce the Problem

1. code

const model = genAI.getGenerativeModel({ model: 'gemini-pro-vision' });
const result = await model.generateContentStream([
        '用中文简体回答',
        ...history,
        {
          inlineData: {
            data: data.base64,
            mimeType: data.mimeType,
          },
        },
      ]);

2.get result like this:
宫保鸡丁是一道著名的川菜，起源于清朝山东巡抚丁宝桢的家厨，以花生、黄瓜、胡萝卜
、������、���丝等配料，再佐以干辣椒、花椒等调味料炒制而成。

3.text contains ���

Specifications

• Version: ^0.1.2
• Platform: windows

This issue was automatically created by Allstar.

Security Policy Violation
Found 5 outside collaborators with admin access.
This policy requires users with this access to be members of the organisation. That way you can easily audit who has access to your repo, and if an account is compromised it can quickly be denied access to organization resources. To fix this you should either remove the user from repository-based access, or add them to the organization.

• Remove the user from the repository-based access. From the main page of the repository, go to Settings -> Manage Access.
  (For more information, see https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-github-user-account/managing-access-to-your-personal-repositories/removing-a-collaborator-from-a-personal-repository)

OR

• Invite the user to join your organisation. Click your profile photo and choose “Your Organization” → choose the org name → “People” → “Invite Member.” (For more information, see https://docs.github.com/en/organizations/managing-membership-in-your-organization/inviting-users-to-join-your-organization)

If you don't see the Settings tab you probably don't have administrative access. Reach out to the administrators of the organisation to fix this issue.

OR

• Exempt the user by adding an exemption to your organization-level Outside Collaborators configuration file.

⚠️ There is an updated version of this policy result! Click here to see the latest update

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Expected Behavior

Actual Behavior

GoogleGenerativeAIError: [404 Not Found]
at makeRequest (file:///D:/git/aily-project/gpt-api/node_modules/@google/generative-ai/dist/index.mjs:205:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async generateContentStream (file:///D:/git/aily-project/gpt-api/node_modules/@google/generative-ai/dist/index.mjs:505:22)
at async GeminiClient.sendMessage (file:///D:/git/aily-project/gpt-api/client/GeminiClient.js:62:24)
at async Object. (file:///D:/git/aily-project/gpt-api/app.js:215:22)
GoogleGenerativeAIError: [404 Not Found]
at makeRequest (file:///D:/git/aily-project/gpt-api/node_modules/@google/generative-ai/dist/index.mjs:205:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async generateContentStream (file:///D:/git/aily-project/gpt-api/node_modules/@google/generative-ai/dist/index.mjs:505:22)
at async GeminiClient.sendMessage (file:///D:/git/aily-project/gpt-api/client/GeminiClient.js:62:24)
at async Object. (file:///D:/git/aily-project/gpt-api/app.js:215:22)

Steps to Reproduce the Problem

        let gemini = new GoogleGenerativeAI(process.env.GEMINI_API_KEY).getGenerativeModel({
            model: "gemini-pro",
            generationConfig: {
                temperature: this.temperature
            }
        });
        const result = await gemini.generateContentStream(conversation.messages);
        for await (const chunk of result.stream) {
            const chunkText = chunk.text();
            console.log(chunkText);
            opts.onProgress(chunkText);
            reply += chunkText;
        }

Specifications

• Version: nodejs v18.16.0
• Platform: windows

Actual Behavior

node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11576:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async makeRequest (/Users/yubowen/Desktop/MyPrivate/node_modules/@google/generative-ai/dist/index.js:195:20)
    at async generateContentStream (/Users/yubowen/Desktop/MyPrivate/node_modules/@google/generative-ai/dist/index.js:519:22)

Node.js v18.18.0

Steps to Reproduce the Problem

1. npm install @google/generative-ai

const { GoogleGenerativeAI } = require("@google/generative-ai");

// Access your API key as an environment variable (see "Set up your API key" above)
const genAI = new GoogleGenerativeAI('xxxxxxxxxxxxxxxx');

async function run() {
  // For text-only input, use the gemini-pro model
  const model = genAI.getGenerativeModel({ model: "gemini-pro" });

  const prompt = "Write a story about a magic backpack."

  const chat = await model.startChat('');
  const result = await chat.sendMessageStream(prompt);
  let text = ''
  for await (const chunk of result.stream) {
    text += chunk.text();
  }
  console.log(text);
}

run();

1. node index.js

Specifications

• Version: Node.js v18.18.0, "@google/generative-ai": "^0.1.3"
• Platform: macOS 12.6.5

Function calling is in Beta release.

You need to specify apiVersion: 'v1beta' when initialising the model.

Please refer to https://github.com/google/generative-ai-js/blob/main/samples/node/advanced-function-calling.js#L52

Originally posted by @alx13 in #66 (comment)

Is there by any chance any examples using function calling in multi-turn conversations? Im sure it can be done, im just finding documentation quite hard to come by and i would love to stop guessing

Any insight would be immensely appreciated

class Agent {
  constructor() {
    this.history = [];
    this.model = genAI.getGenerativeModel({ model: "gemini-pro", tools, generationConfig, safetySetting }, { apiVersion: "v1beta" });
    this.chat = model.startChat({ history: this.history });
    this.browser = new AIB();
  }

  async runFunction(name, args) {
    const fn = functions[name];
    if (!fn) {
      throw new Error(`Unknown function "${name}"`);
    }

    const fr = {
      role: "function",
      parts: [
        {
          functionResponse: {
            name,
            response: {
              name,
              content: await functions[name](args)
            }
          },
        }
      ]
    };

    this.history.push(fr);
    return fr;
  }


  async msg(query) {
    let result = await this.chat.sendMessage(query);
    if(result?.response?.candidates[0]?.content) {
      let cont = false;
      if(result?.response?.candidates[0]?.content?.parts[0]?.functionCall) {
        cont = true;
        while(cont) {
          let { name, args } = result.response.candidates[0].content.parts[0].functionCall;
          let response = await this.runFunction(name, args);
          result = await this.chat.sendMessage([response]);
          if(!result?.response?.candidates[0]?.content?.parts[0]?.functionCall) {
            cont = false;
          }
        }
      }
    }
  }
}

GoogleGenerativeAIError: [400 Bad Request] Invalid JSON payload received. Unknown name "role" at 'contents[5].parts[0]': Cannot find field.
Invalid JSON payload received. Unknown name "parts" at 'contents[5].parts[0]': Cannot find field. [{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"contents[5].parts[0]","description":"Invalid JSON payload received. Unknown name "role" at 'contents[5].parts[0]': Cannot find field."},{"field":"contents[5].parts[0]","description":"Invalid JSON payload received. Unknown name "parts" at 'contents[5].parts[0]': Cannot find field."}]}]

or if possibly someone notices where im going wrong here 🙏🏼

It'd be great if you could add support for the list_models API https://ai.google.dev/api/python/google/generativeai/list_models.

This works:

const genAI = new GoogleGenerativeAI(apiKey);
const genModel = genAI.getGenerativeModel({ 'gemini-pro' });
const result = await genModel.generateContent(content);

But when I stream, I get the error TypeError: response.body.pipeThrough is not a function:

const genAI = new GoogleGenerativeAI(apiKey);
const genModel = genAI.getGenerativeModel({ 'gemini-pro' });
//error occurs here
const result = await genModel.generateContentStream(content);

I am using node v21.3.0
npm version @google/generative-ai : "^0.1.3"

Thanks in advance!

This issue was automatically created by Allstar.

Security Policy Violation
Dismiss stale reviews not configured for branch main

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

My understanding is that the library requires apiVersion to be set to v1beta for Gemini 1.5 to work. However, the apiVersion is not passed down to the implementation of countTokens, due to which it makes a request at the older endpoint.

This is with v0.6.0 (latest)

Actual Behavior

const modelInstance = googleAI.getGenerativeModel(
    { model: "gemini-1.5-pro-latest" },
    { apiVersion: "v1beta" },
  )

modelInstance.generateContent()
// this sends a request to https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro-latest:generateContent
// ...

modelInstance.countTokens
// this sends a request to https://generativelanguage.googleapis.com/v1/models/gemini-1.5-pro-latest:countTokens()
// which errors out

I believe the issue lies here (requestOptions are not passed): https://github.com/google/generative-ai-js/blob/79b7651547536183108f0b78791761d1ece01f88/packages/main/src/models/generative-model.ts#L156

Expected Behavior

The countTokens method should make a request to the v1beta endpoint

As mentioned in #56 (comment), countTokens() is not passing RequestOptions to RequestUrl.

We are trying to make a sql query generator with chat complation.

prompt: Show me the customers

history:

[
  {"role": "user", "parts": "### SYSTEM CONTEXT ### \nYou are an *MSSQL* command generator. Use table schema for generating sql commands. ### EXAMPLE RESONSE ### \`\`\`sql ... \`\`\`  \n ### DATABASE SCHEME ### ......."},
  {"role": "model", "parts": "OK" }
]

const gemini = gai.getGenerativeModel({ model });
const chat = gemini.startChat({history, generationConfig: { temperature: 0.2 }});

const result = await chat.sendMessage(prompt);
const response = await result.response;

It gives me cool sql commands but some table names are not logical names (like crnt_tbl for current table). AI tries to access tables that do not exist in the schema, causing the query to not work.

I want it to only use existing table names in the schema. How can I set this up?

git push --follow-tags doesn't seem to be effective in pushing the new tag to github. May have to push the tag by name.

Expected Behavior

sendMessageStream with Chinese prompt should returns right chunks.

Actual Behavior

sendMessageStream with Chinese prompt returns malfunctioning chunks which contains ���.

Steps to Reproduce the Problem

1. const prompt = '创作一首古诗';
   const chat = await model.startChat(history);
   const result = await chat.sendMessageStream(prompt);
   let text = ''
   for await (const chunk of result.stream) {
     text += chunk.text();
   }
   console.log(text)

2. get result like this:

    **寒夜**
    
    月寒山冷夜漫长，
    秋声���瑟送寒凉。
    银辉霜雪遍地洒，
    红叶舞尽染霜霜。
    
    寒风凛冽凌人骨，
    万籁俱寂月霜寒。
    古刹钟声伴孤影，
    一曲悲歌满心酸。
    
    寒江衰柳起愁思，
    霜梅雪月亦难禁。
    孤篇残卷叹今昔，
    满目疮痍叹世事。
    
    寒������滚������来，
    哀鸿遍野亦不悔。
    纵然前路多险阻，
    岂能轻言半途弃？
   

3. text contains ���

Specifications

• Version: ^0.1.1
• Platform: macOS

Expected Behavior

System instructions enable users to steer the behavior of the model based on their specific needs and use cases. When you set a system instruction, you give the model additional context to understand the task, provide more customized responses, and adhere to specific guidelines over the full user interaction with the model. For developers, product-level behavior can be specified in system instructions, separate from prompts provided by end users.

You can use system instructions in many ways, including:

Defining a persona or role (for a chatbot, for example)
Defining output format (Markdown, YAML, etc.)
Defining output style and tone (for example, verbosity, formality, and target reading level)
Defining goals or rules for the task (for example, returning a code snippet without further explanations)
Providing additional context for the prompt (for example, a knowledge cutoff)
When a system instruction is set, it applies to the entire request. It works across multiple user and model turns when included in the prompt. System instructions are part of your overall prompts and therefore are subject to standard data use policies.

Actual Behavior

no System instructions

Expected Behavior

The File api is only available in the python sdk: https://ai.google.dev/tutorials/prompting_with_media

Actual Behavior

To upload file in js sdk and use it with gemini-1.5-pro

Steps to Reproduce the Problem

none, this is a feature quest

Specifications

• Version: 0.5.0
• Platform: web

Expected Behavior

The code on this page should work.

Actual Behavior

TypeError: Cannot use 'in' operator to search for 'text' in H
    at validateChatHistory ([...]@google/generative-ai/dist/index.mjs:758:25)

Steps to Reproduce the Problem

Run the code from the sample:

const { GoogleGenerativeAI } = require("@google/generative-ai");

// Access your API key as an environment variable (see "Set up your API key" above)
const genAI = new GoogleGenerativeAI(process.env.API_KEY);

async function run() {
  // For text-only input, use the gemini-pro model
  const model = genAI.getGenerativeModel({ model: "gemini-pro"});

  const chat = model.startChat({
    history: [
      {
        role: "user",
        parts: "Hello, I have 2 dogs in my house.",
      },
      {
        role: "model",
        parts: "Great to meet you. What would you like to know?",
      },
    ],
    generationConfig: {
      maxOutputTokens: 100,
    },
  });

  const msg = "How many paws are in my house?";

  const result = await chat.sendMessage(msg);
  const response = await result.response;
  const text = response.text();
  console.log(text);
}

run();

Specifications

• Version: @google/generative-ai": "^0.3.0
• Platform: node

Specifically, I believe this was broken by #32. It would be trivial to make that change backwards-compatible by having validateChatHistory branch on whether parts is a string or array (or otherwise error), but either way the documentation should be updated.

Description of the bug:

// npm install @google/generative-ai

const {
  GoogleGenerativeAI,
  HarmCategory,
  HarmBlockThreshold,
} = require("@google/generative-ai");

const MODEL_NAME = "tunedModels/<modelname>";
const API_KEY = "YOUR_API_KEY";

async function run() {
  const genAI = new GoogleGenerativeAI(API_KEY);
  const model = genAI.getGenerativeModel({ model: MODEL_NAME });

  const generationConfig = {
    temperature: 0.9,
    topK: 1,
    topP: 1,
    maxOutputTokens: 8192,
  };

  const safetySettings = [
  ];

  const parts = [
    {text: "input: "},
    {text: "output: "},
  ];

  const result = await model.generateContent({
    contents: [{ role: "user", parts }],
    generationConfig,
    safetySettings,
  });

  const response = result.response;
  console.log(response.text());
}

run();```

### Actual vs expected behavior:


Current behavior the URL is pointing to ```v1/tunedModels``` but for tuned models it should point to ```v1beta/tunedModels``` . There is no option found to change this URL endpoint 

### Any other information you'd like to share?


_No response_

Expected Behavior

No API key in error message

Actual Behavior

Just see where url comes from in that line: https://github.com/google/generative-ai-js/blob/7d119446081bb193c3178fd285d64950d6627645/packages/main/src/requests/request.ts#L88

Specifications

• Version: 0.1.1 on Linux
• Platform: Node 18

Expected Behavior

Trying to run the example here: https://github.com/google/generative-ai-js/tree/main?tab=readme-ov-file#try-out-a-sample-app

Actual Behavior

node:internal/process/promises:246
          triggerUncaughtException(err, true /* fromPromise */);
          ^

ReferenceError: fetch is not defined
    at makeRequest (file:///Users/logan/code/generative-ai-js/samples/node/node_modules/@google/generative-ai/dist/index.mjs:193:9)
    at countTokens (file:///Users/logan/code/generative-ai-js/samples/node/node_modules/@google/generative-ai/dist/index.mjs:755:28)
    at GenerativeModel.countTokens (file:///Users/logan/code/generative-ai-js/samples/node/node_modules/@google/generative-ai/dist/index.mjs:853:16)
    at run (file:///Users/logan/code/generative-ai-js/samples/node/simple-text.js:26:39)
    at file:///Users/logan/code/generative-ai-js/samples/node/simple-text.js:35:1
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:281:24)
    at async loadESM (node:internal/process/esm_loader:88:5)
    at async handleMainPromise (node:internal/modules/run_main:65:12)
logan@oai-logan node % ls -l

Steps to Reproduce the Problem

Walked through the steps here: https://github.com/google/generative-ai-js/tree/main?tab=readme-ov-file#try-out-a-sample-app

Specifications

• Version: Node v16.13.0
• Platform: MacOS

Expected Behavior

It works

Actual Behavior

[GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1/models/gemini-1.5-pro-latest:countTokens: [404 Not Found] models/gemini-1.5-pro-latest is not found for API version v1, or is not supported for countTokens. Call ListModels to see the list of available models and their supported methods.

Likely we need to change v1 to v1beta somewhere before making requests to countTokens endpoint (the url above is still v1)

Steps to Reproduce the Problem

const googleModel = this.client.getGenerativeModel({ model: 'gemini-1.5-pro-latest' }, { apiVersion: 'v1beta' })

await googleModel.countTokens(chunks) // error thrown here

Specifications

• Version: 7.0.1
• Platform: macOS

I don't see function calling support for the Node SDK. Is it coming soon? It's a bit inconvenient to maintain native SDK and raw fetch code at the same time!

Expected Behavior

model gemini-pro startChat with traditional Chinese prompt returss result text.

Actual Behavior

model gemini-pro startChat with traditional Chinese prompt returns empty string.

Steps to Reproduce the Problem

1. asked the question in traditional Chinese

  const model = genAI.getGenerativeModel({ model: "gemini-pro"});
  
  const chat = model.startChat({
    history: [
    ],
    generationConfig: {
      maxOutputTokens: 100,
    },
  });

  const msg = "創作一首古詩";

  const result = await chat.sendMessage(msg);
  const response = await result.response;
  const text = response.text();
  console.log(text);
  console.log(text.length);

2. get result

0

3. If I asked the same question in simplified Chinese

  const model = genAI.getGenerativeModel({ model: "gemini-pro"});
  
  const chat = model.startChat({
    history: [
    ],
    generationConfig: {
      maxOutputTokens: 100,
    },
  });


  const msg = "创作一首古诗";

  const result = await chat.sendMessage(msg);
  const response = await result.response;
  const text = response.text();
  console.log(text);
  console.log(text.length);

4. the result like this:

Specifications

**清溪**

溪水潺潺绕石鸣，
绿柳依依拂清风。
游鱼戏藻影绰绰，
鸟语花香沁心胸。

山岚缥缈似烟雾，
掩映山色醉人目。
我醉青山不知归，
人间烦恼尽消疏。
80

• Version: 0.2.1
• Platform: ubuntu 22.04 in WSL

Will there be support to use video data like in the @google-cloud/vertexai library?

I have such setting when calling the API
exports.gemProVision = new GoogleGenerativeAI( process.env.GEN_AI_API_KEY ).getGenerativeModel({ model: 'gemini-pro-vision', safety_settings: [ { category: HarmCategory.HARM_CATEGORY_UNSPECIFIED, threshold: HarmBlockThreshold.BLOCK_NONE, }, { category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, threshold: HarmBlockThreshold.BLOCK_NONE, }, { category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, threshold: HarmBlockThreshold.BLOCK_NONE, }, { category: HarmCategory.HARM_CATEGORY_HARASSMENT, threshold: HarmBlockThreshold.BLOCK_NONE, }, { category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT, threshold: HarmBlockThreshold.BLOCK_NONE, }, ], })

However I get the following:
imageResp.response: {"promptFeedback":{"blockReason":"SAFETY","safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"MEDIUM"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}}

And sometimes it returns:
imageResp.response: {"promptFeedback":{"blockReason":"OTHER"}}

I wonder what's the use of threshold: HarmBlockThreshold.BLOCK_NONE here?

Expected Behavior

Ability to specify API_VERSION (for example: "v1beta") as a configuration.

Actual Behavior

Currently the API_VERSION is hardcoded in the package to "v1" and there is no way to change it. For example, "v1beta" has functionality, such as function calling, that is currently inaccessible due to this hardcoded value.

Can we use a system role in chat history to provide instructions to model?

For example:

{
"role": "system",
"parts": "You are an sql generator and your platform is MSSQL."
}

this doesn't appear to grant me access to the file methods Prompting with media files and here File prompting strategies

  const model = genAI.getGenerativeModel({ model: MODEL_NAME},{apiVersion: 'v1beta',});

Again I'm trying to get support for the file upload beta endpoint. I decided to try to load it though the discovery doc. I can load the discovery doc and call the file endpoint

require('dotenv').config();

const API_KEY = process.env.API_KEY; // Get the api key from env
const MODEL_NAME = process.env.TEXT_MODEL_NAME_LATEST; // Get the model name from env

// Importing the GoogleGenerativeAI class from the "@google/generative-ai" package
const { GoogleGenerativeAI } = require("@google/generative-ai");

// Access your API key as an environment variable (see "Set up your API key" above)
const genAI = new GoogleGenerativeAI(process.env.API_KEY);

async function run() {
    // For text-only input, use the gemini-pro model
    const model = genAI.getGenerativeModel({ model: MODEL_NAME},{apiVersion: 'v1beta',});

    const prompt = "Write a story about a magic backpack."

    const result = await model.generateContent(prompt);
    const response = await result.response;
    const text = response.text();
    console.log(text);
}

run();

Its resulting in an error as its not adding the api key.

 errors: [
    {
      message: "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.",
      domain: 'global',
      reason: 'forbidden'
    }
  ],

Any one know how to add the api key to this?

cross posted How to add an api key to discovery loaded beta endpoint for the google js client library

The sendMessageStream methods in the ChatSession class do not handle promise rejections, potentially leading to unhandled promise rejections if generateContentStream fail. The internal promise chain does not have a .catch() to handle these errors, and they are not propagated to be catchable by the caller's try-catch block.

https://github.com/google/generative-ai-js/blob/7d119446081bb193c3178fd285d64950d6627645/packages/main/src/methods/chat-session.ts#L140

Expected Behavior

Check below screenshots:

This is taken from the example provided in the docs. Type information seems to be setup incorrectly.

File:
https://github.com/google/generative-ai-js/blob/2be48f8e5427f2f6191f24bcb8000b450715a0de/packages/main/types/responses.ts#L32

Actual Behavior

here we will need to do something like this:

export declare interface GenerateContentResult {
    response: Promise<EnhancedGenerateContentResponse>;
}

Or update the docs and change:

const response = await result.response;

to

const response = result.response;

this seems to be more likely as GenerateContentResult has been used without await at many places, which I believe is the expected behaviour.

Steps to Reproduce the Problem

1. Use the example from docs
2. Notice the incorrect type information setup

Specifications

• Version: 0.1.3
• Platform: Not required

Expected Behavior

model.sendMessageStream(msg) to be implemented as it is mentioned in the documentation here:
https://ai.google.dev/tutorials/node_quickstart?hl=en

Actual Behavior

Not implemented.

• Version: 0.3.0
• Platform: iOS

Hey, I'm having an issue within the demo for https://ai.google.dev/tutorials/node_quickstart#generate-text-from-text-input. I'm not able to run the demo in Nodejs.

Output:
The error was: ReferenceError: fetch is not defined
at makeRequest Nodejs/node_modules/@google/generative-ai/dist/index.js:187:9)
at generateContent (Nodejs/node_modules/@google/generative-ai/dist/index.js:511:28)
at GenerativeModel.generateContent (Nodejs/node_modules/@google/generative-ai/dist/index.js:785:16)
at run (Nodejs/controllers/bard-controller.js:18:30)
at Object. (Nodejs/controllers/bard-controller.js:24:1)
at Module._compile (internal/modules/cjs/loader.js:1063:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
at Module.load (internal/modules/cjs/loader.js:928:32)
at Function.Module._load (internal/modules/cjs/loader.js:769:14)
at Module.require (internal/modules/cjs/loader.js:952:19)

Code from my end

const { GoogleGenerativeAI } = require("@google/generative-ai");

// Access your API key as an environment variable (see "Set up your API key" above)
const genAI = new GoogleGenerativeAI(API_KEY);

async function run() {
// For text-only input, use the gemini-pro model
const model = genAI.getGenerativeModel({ model: "gemini-pro"});

const prompt = "Write a story about a magic backpack."

const result = await model.generateContent(prompt);
const response = await result.response;
const text = response.text();
console.log(text);
}

On this page:

https://ai.google.dev/api/rest/v1beta/models/generateAnswer

In the Response Body section, there is this text for the answer field:

"Candidate answer from the model.

Note: The model always attempts to provide a grounded answer, even when the answer is unlikely to be answerable from the given passages. In that case, a low-quality or ungrounded answer may be provided, along with a low answerableProbability."

While it is true that you will always get a Candidate answer object back, in the ABSTRACTIVE and VERBOSE answer styles, where the model is forced to synthesize text, you you will not get a content child object field containing answer text. When the answerableProbabillity is too low, you will only get a content child object with answer text in the EXTRACTIVE answer style, since the model just has to produce the grounding answer that is the closest match and does not have to synthesize any text to serve as a reply. So if you use the word "answer" in the context of a Candidate object, as the docs use it, then the docs are correct. But in the traditional sense of an answer containing text that replies to a question, the doc text is less helpful then it could be.

I suggest mentioning this in the docs, because I spent a fair amount of time trying to figure out why I didn't get any usable answer text due to this model behavior, until I wrote a rigorous test harness that tries the exact same user query and inline grounding passages set with each of the 3 AnswerStyle values. A short doc note about this could save other developers some confusion.

Has function calling been implemented? I can see the type definitions in the package but it doesn't seem to be working. I receive this error:

[GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent: [400 ] Invalid JSON payload received. Unknown name "tools": Cannot find field. [{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"description":"Invalid JSON payload received. Unknown name "tools": Cannot find field."}]}]

I highly recommend looking at the "runTools" method in the openai npm package. It makes functions calling very intuitive with many useful helpers. It adds significant improvements over the traditional function calling approach.

Mainly that, in the traditional approach, to receive a streaming response from a function call, the user needs to manually disable function calling since the chat.completion is unable to stream a response when calling a function. In order to get a streaming response, the client needs to first call the functions, get a response and then call the api again, with the function response, to get a streaming response from the model.

"runTools" solves this attaching helper events to the method.

Expected Behavior

respond to messages in a word-for-word or line-by-line manner

Actual Behavior

Getting errors or always responding to all messages at once

Steps to Reproduce the Problem

1.I was trying to return an event stream using Koa and ran into an error. Here's a snippet of the code from the Node.js route:

  import { PassThrough, pipeline } from 'stream'
  const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
  const model = genAI.getGenerativeModel(modelParams)
  const chat = model.startChat(startChatParams) 
  const sseStream = new PassThrough({ objectMode: true })
  const res = await chat.sendMessageStream(prompt)
  pipeline(res.stream, sseStream);
  ctx.body = sseStream

This is the error message after running:

The "streams[stream.length - 1]" property must be of type function. Received an instance of PassThrough

2. I tried responding in a different way:

import { Transform } from 'stream'
class SSEStream extends Transform {
  constructor() {
    super({
      writableObjectMode: true
    });
  }
  _transform(data: any, _encoding: any, done: () => void) {
    try {
      this.push(`data: ${JSON.stringify(data)}\n\n`);
      done();
    } catch (error) {
      done();
    }
  }
}
  const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
  // ...
  const res = await chat.sendMessageStream(prompt)
  const sseStream = new SSEStream()
  ctx.body = sseStream
  for await (const chunk of res.stream) {
    const chunkText = chunk.text();
    sseStream.write(chunkText);
  }

The client will only receive the final complete message once, instead of getting it line by line. Here's a snippet of the front-end code:

const response = await fetch();
if(response.body) {
    const reader = response.body.getReader()
    const decoder = new TextDecoder()
    let fullText = ''
    while (true) {
        const { value, done } = await reader.read()
        if (done) {
            break
        } else {
            const chunk = decoder.decode(value)
            fullText += chunk
            if (onText) {
                onText({ text: fullText, cancel })
            }
        }
    }
}

Specifications

• Version:
  @google/generative-ai: 0.1.3
  node: 20.10.0

Hi,

Seems like tokeCount is missing in the API.
https://ai.google.dev/api/rest/v1/GenerateContentResponse#Candidate

Any reason why?

Thank you!

Actual Behavior

The docs mention that you can pass model parameters to genAI.getGenerativeModel.getGenerativeModel:

const generationConfig = {
  stopSequences: ["red"],
  maxOutputTokens: 200,
  temperature: 0.9,
  topP: 0.1,
  topK: 16,
};

const model = genAI.getGenerativeModel({ model: "MODEL_NAME",  generationConfig });

But if you pass the parameters there, they are silently ignored when using model.startChat API. You need to pass these parameters directly to startChat, even though it is already a method of GenerativeModel.

I confirmed that by making the requests through a proxy (https://httptoolkit.com/).

The same applies for safetySettings.

Expected Behavior

model.startChat should inherit generationConfig, safetySettings, etc., from model.

Specifications

• Version: 0.6.0
• Platform: Windows

Not really, an issue, but a help request!

I used the Google Generative AI Sample for Web https://github.com/google/generative-ai-js/tree/main/samples/web on a live HTTP server. I placed the application at https://www.example.com/genai, where genai is the folder for the app, and now I need assistance in setting the API Key as an environment variable according to the instructions provided, see https://github.com/google/generative-ai-js/blob/main/samples/web/http-server.js ? Please note that I do not have root access.

However, any help to transform how the API Key is accessed just like it is done for ChatGPT https://niek.github.io/chatgpt-web? I mean to make it possible to just copy and paste the key on the web interface rather than storing it in the back-end?

I cant use "await chat.sendMessageStream(newmessage.text)" in react native app. It gives
"[TypeError: Cannot read property 'pipeThrough' of undefined] " error.

"await chat.sendMessage(newmessage.text)" works fine.

const genAI = new GoogleGenerativeAI(API_KEY);
const model = genAI.getGenerativeModel({ model: "gemini- 
pro",safetySettings:safetySettings,generationConfig:generationConfig});
const chat = model.startChat({safetySettings:safetySettings,history:history.current})
const result = await chat.sendMessage(newmessage.text).catch((e)=>{
    console.log(e)
  })
  let text = '';
for await (const chunk of result.stream) {
const chunkText = chunk.text();
console.log(chunkText);
text += chunkText;
}

  console.log(text);

Expected Behavior

User has access to tuned models after using Application Default Credentials (ADC) and exporting default credentials

Actual Behavior

User does not have an access to a model. Request failed with error Error fetching from https://generativelanguage.googleapis.com/v1/tunedModels/MODEL_NAME:generateContent: [403 Forbidden] You do not have permission to access tuned model MODEL_NAME

Steps to Reproduce the Problem

1. Follow this steps https://ai.google.dev/docs/oauth_quickstart
2. Export path as GOOGLE_APPLICATION_CREDENTIALS
3. Create new generative model with tuned model

const model = this.ai.getGenerativeModel({
  model: "tunedModels/MODEL_NAME",
})

4. Use tuned model as usually

await model.generateContent(`Some text`);

Specifications

• Version: 0.6.0
• Platform: MacOS

Expected Behavior

Some regions can't access the API directly, so it's cause fetching error.

Actual Behavior

Is it possible to add proxy when the model is sending message to google AI API endpoint?
Thanks

Steps to Reproduce the Problem

Specifications

• Version:
• Platform:

Right now, if the user calls sendMessage or sendMessageStream before the last exchange has completed, we just put it in a queue (promise chain) and send it when the last response completes. This can cause unexpected output, as the user perceives that they sent two messages in a row with no response in between (history is user-user-model), whereas the backend actually responded to the first user message before seeing the second one (user-model-user). We should throw an error (or no-op and warn) to let the user know they are trying to send a message before the last response completed, and may get unexpected results.

Expected Behavior

I setup all 4 safety settings to HarmBlockThreshold.BLOCK_NONE, I'm expecting a valid response

Actual Behavior

I got error:

sendMessage() was unsuccessful. Response was blocked due to OTHER. Inspect response object for details.
response: { promptFeedback: { blockReason: 'OTHER' }, text: [Function (anonymous)], functionCall: [Function (anonymous)], functionCalls: [Function (anonymous)] }

I'm getting same error both in Google AI Studio, and in my nodejs program using api

note, api call thinks that parameter is illegal, even though I see that category is defined in source code:
{ category: HarmCategory.HARM_CATEGORY_UNSPECIFIED, threshold: HarmBlockThreshold.BLOCK_NONE, },

Steps to Reproduce the Problem

1. const safetySettings = [ { category: HarmCategory.HARM_CATEGORY_HARASSMENT, threshold: HarmBlockThreshold.BLOCK_NONE, }, { category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, threshold: HarmBlockThreshold.BLOCK_NONE, }, { category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT, threshold: HarmBlockThreshold.BLOCK_NONE, }, { category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, threshold: HarmBlockThreshold.BLOCK_NONE, }, ];
2. const apiVersion = 'v1beta';
3. const generationConfig = { temperature: 0.9,};

4. `const model = genAI.getGenerativeModel({ model: "gemini-1.5-pro-latest" , generationConfig, safetySettings }, {apiVersion});`
   

5. `const result = await chat.sendMessage("translate following to English: Opere meme Ca tanısıyla izlenen hastada sağ meme mastektomizedir. Mastektomi lojunda belirgin kitle lezyonu saptanmadı.");`
   

Specifications

• Version: "@google/generative-ai": "^0.7.1",
• Platform: Windows 10, nodejs v20.12.2 (LTS)

I am able to use generateContent() for text completions without problem. However, I can't seem to find the generateAnswer() method for use with the Attributed Question-Answer generative AI model (i.e. = models\aqa). In fact, the Tasks enum doesn't even have an value for that:

export enum Task {
  GENERATE_CONTENT = "generateContent",
  STREAM_GENERATE_CONTENT = "streamGenerateContent",
  COUNT_TOKENS = "countTokens",
  EMBED_CONTENT = "embedContent",
  BATCH_EMBED_CONTENTS = "batchEmbedContents",
}

Can someone tell me where it is? Note, this is not a priority because I already added my own code to do this with the REST API, but at some point I would like to put everything through the generate-ai-js package instead of just the generateContent requests.

Reading the docs I can't find a section where specifies the possible API error codes, in the examples/references it would be helpful to know what are the possible errors the API can return for error handling.

I hope this can be implemented, thanks!.