Multimodal AI API Integration with Gemini 3.6 and GPT-5.6 in Node.js

Multimodal AI API integration in Node.js has two compact server-side routes through Gemini’s Interactions API and OpenAI’s Responses API. I installed the latest SDK releases on Node.js 26.5.1, exercised both request adapters with a PNG fixture, and reached each provider endpoint to verify the authentication boundary.

What each API accepts in July 2026

Google’s catalog lists Gemini 3.6 Flash for text, image, video, audio, and Portable Document Format (PDF) input, while OpenAI’s vision documentation uses GPT-5.6 for image URLs, Base64 image data, and uploaded file IDs.

The APIs overlap on image analysis, but they are not interchangeable for every modality because OpenAI’s documented Responses API route accepts images rather than direct video, while Gemini can accept a video through its Files API or as inline data within the documented size limit.

Choose Gemini when direct video or audio understanding changes the product. Choose OpenAI when your service already uses the Responses API and image analysis is the required modality.

Set up the Node.js project

Install the official provider packages without a version pin so a new project resolves the latest stable releases available to npm.

npm install @google/genai openai

Keep GEMINI_API_KEY and OPENAI_API_KEY in the server environment. Do not place either key in browser JavaScript, commit it to Git, or accept it from an untrusted request body.

The SDK constructors read their provider variables by default. A server route can call the adapter without copying either credential into application code.

Build one adapter for both providers

The module below reads a PNG once, converts it to Base64, and creates the request shape each SDK expects. Dependency injection through the optional client argument lets the same functions run against controlled test clients.

import fs from "node:fs";
import { pathToFileURL } from "node:url";
import OpenAI from "openai";
import { GoogleGenAI } from "@google/genai";

export async function describeWithGemini({ imagePath, prompt, client }) {
  const imageData = fs.readFileSync(imagePath, { encoding: "base64" });
  const api = client ?? new GoogleGenAI({});
  const interaction = await api.interactions.create({
    model: "gemini-3.6-flash",
    input: [
      { type: "text", text: prompt },
      { type: "image", data: imageData, mime_type: "image/png" },
    ],
  });
  return interaction.output_text;
}

export async function describeWithOpenAI({ imagePath, prompt, client }) {
  const imageData = fs.readFileSync(imagePath, { encoding: "base64" });
  const api = client ?? new OpenAI();
  const response = await api.responses.create({
    model: "gpt-5.6",
    input: [
      {
        role: "user",
        content: [
          { type: "input_text", text: prompt },
          {
            type: "input_image",
            image_url: `data:image/png;base64,${imageData}`,
          },
        ],
      },
    ],
  });
  return response.output_text;
}

async function main() {
  const [provider, imagePath, ...words] = process.argv.slice(2);
  const prompt = words.join(" ") || "Describe this image in one sentence.";

  if (!provider || !imagePath) {
    console.error("Usage: node multimodal.mjs <gemini|openai> <image.png> [prompt]");
    process.exitCode = 1;
    return;
  }

  const describe = provider === "gemini"
    ? describeWithGemini
    : provider === "openai"
      ? describeWithOpenAI
      : null;

  if (!describe) {
    throw new Error(`Unknown provider: ${provider}`);
  }

  console.log(await describe({ imagePath, prompt }));
}

if (import.meta.url === pathToFileURL(process.argv[1]).href) {
  await main();
}

Run the module with a provider name, an image path, and an optional prompt. Both commands keep the image on the server until the chosen SDK sends the request.

node multimodal.mjs gemini fixture.png "Describe the image in one sentence."
node multimodal.mjs openai fixture.png "Describe the image in one sentence."

Test the request boundary before spending API credits

A useful adapter test checks the model identifier, modality names, MIME type, encoded image, and response field. It should also reject an unsupported provider before any network call begins.

import assert from "node:assert/strict";
import fs from "node:fs";
import test from "node:test";
import { describeWithGemini, describeWithOpenAI } from "./multimodal.mjs";

const imagePath = new URL("./fixture.png", import.meta.url);

const onePixelPng = Buffer.from(
  "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Y9Z4RQAAAAASUVORK5CYII=",
  "base64",
);
fs.writeFileSync(imagePath, onePixelPng);

test("Gemini adapter sends text and inline image data", async () => {
  let captured;
  const client = {
    interactions: {
      create: async (request) => {
        captured = request;
        return { output_text: "A white square." };
      },
    },
  };

  const output = await describeWithGemini({
    imagePath,
    prompt: "Describe the image.",
    client,
  });

  assert.equal(output, "A white square.");
  assert.equal(captured.model, "gemini-3.6-flash");
  assert.deepEqual(captured.input.map((part) => part.type), ["text", "image"]);
  assert.equal(captured.input[1].mime_type, "image/png");
});

test("OpenAI adapter sends a Responses API image input", async () => {
  let captured;
  const client = {
    responses: {
      create: async (request) => {
        captured = request;
        return { output_text: "A white square." };
      },
    },
  };

  const output = await describeWithOpenAI({
    imagePath,
    prompt: "Describe the image.",
    client,
  });

  assert.equal(output, "A white square.");
  assert.equal(captured.model, "gpt-5.6");
  assert.deepEqual(
    captured.input[0].content.map((part) => part.type),
    ["input_text", "input_image"],
  );
  assert.match(captured.input[0].content[1].image_url, /^data:image\/png;base64,/);
});

test("the CLI rejects an unknown provider before any API call", async () => {
  const { spawnSync } = await import("node:child_process");
  const result = spawnSync(
    process.execPath,
    ["multimodal.mjs", "other", "fixture.png"],
    { cwd: new URL(".", import.meta.url), encoding: "utf8" },
  );
  assert.notEqual(result.status, 0);
  assert.match(result.stderr, /Unknown provider/);
});

Run the tests with Node’s built-in test runner.

node --test multimodal.test.mjs

The run completed three tests with no failures on Node.js 26.5.1, @google/genai 2.14.0, and openai 7.1.0.

I also sent each SDK request with an intentionally invalid temporary credential, and the provider endpoints rejected them with HTTP 400 and 401 responses.

Those checks prove request assembly, SDK routing, response extraction, and the authentication boundary, but they do not prove model quality, latency, account access, or billable inference because no provider API key was available in the execution environment.

Handle image, video, and file size deliberately

Base64 is convenient for small images because one request contains the prompt and media. It also expands the payload, so it should not become the default for long clips or reusable assets.

Google’s image documentation caps the total inline request at 20 MB and recommends the Files API for larger or reused images, while its video documentation recommends the Files API for files above 100 MB, long-form clips, or any video you plan to prompt more than once.

• Validate the MIME type from decoded bytes rather than trusting the filename.
• Reject oversized uploads before reading them into a Node.js Buffer.
• Upload a reusable Gemini asset once, then store its file URI for later requests.
• Extract representative frames before sending video to an image-only API route.

Frame extraction changes the question the model can answer. Sparse frames may catch objects and text but lose motion, timing, and audio context, so direct Gemini video input is the stronger fit when temporal sequence matters.

Return structured application data

Production code often needs fields that another service can validate, so ask for a small JSON object, validate it locally, and reject a response that does not match the contract.

Do not treat schema compliance as factual accuracy. Structured output controls the response shape, while tests and human review must still check whether the visual claim matches the source image.

If you are exposing several providers behind one route, my JavaScript AI wrapper walkthrough shows how to separate provider code from the public handler. That boundary also gives you one place to add timeouts, retries, and usage logs.

Control cost without guessing

Images count toward provider usage, and image detail or resolution can change token consumption. Measure usage metadata from completed calls instead of estimating spend from file size alone.

Start with the cheapest model that meets your accuracy threshold, then route uncertain cases to a stronger model. The AI API cost guide covers caching, model routing, and request logging without coupling those controls to one vendor.

Protect visual data before upload

Screenshots and photos can contain email addresses, access tokens, faces, location clues, and customer records outside the intended subject. Redact sensitive regions before the provider request, then delete temporary media on a short retention schedule.

• Authorize the user before reading the source file.
• Strip metadata when camera location and device details are unnecessary.
• Record provider, model, purpose, and deletion time in an audit log.
• Keep media from different users in separate storage namespaces.
• Require human review before a visual classification triggers a high-impact action.

A multimodal model can misread small text, rotated objects, counts, or spatial relationships. The strongest developer objection is valid here: passing an adapter test does not make visual output safe for medical, financial, identity, or access-control decisions.

Frequently asked questions

These boundaries cover the implementation decisions that usually surface after the first successful request.

A useful next move

Run the local adapter tests first, then add one provider key in a staging environment and compare both outputs against the same image set. Keep the provider whose errors, latency, modality support, and usage fit the workload rather than choosing from a model name alone.

The inspectable artifact is the request adapter and its test suite. Extend that suite with malformed files, oversized uploads, provider timeouts, and responses that fail your application schema before the endpoint reaches users.