AI API Gateway FAQs | University IT

Below is a list of expected questions and answers for the AI API Gateway.

Basics

What is the AI API Gateway?

The AI API Gateway is a centralized service providing direct API access to the advanced AI models featured in the Stanford AI Playground. The AI API Gateway uses the opensource LiteLLM platform to help enable teams to develop AI tools and integrate AI capabilities into your applications.

What models are accessible through the AI API Gateway?

The AI API Gateway has access to all of the major Large Language Models available in the AI Playground. View the AI API Gateway Rates page for a complete list of the currently available models.

Image generation models are expected to be supported soon.

Getting started

How do I request an API key?

You can use the Add AI API Gateway Key form to request the creation of a new API key.

How do I get started with my API key?

Once you receive an API key, follow the guidance in the AI Playground Access and Usage help page.

I requested a key from the ticket, but haven’t received the key yet. How long does it take to generate an API key?

It generally takes two to five days to complete all the setup for the request, assuming that the PTA owner has approved.

Before opening a new ticket:

If your ticket has not been completed, please check that the PTA approver you selected has approved the request before opening a ticket.
If the PTA owner has approved, and you have not received the API key within five business days, then please add a comment to your original ticket.

If the ticket is closed and still you have not received your key, then please open a ticket to inquire about the issue.

Cost and expense

How much does it cost to use the AI API Gateway?

The AI API Gateway is a chargeback service based on usage. Rates are determined by our vendors and the negotiated Stanford discount. Please note that the vendors change these prices often.

For detailed pricing rates, please refer to the AI API Gateway Rates page. During your service setup, you will be asked to indicate a max budget, which will prevent API calls beyond that budget.

How does billing work?

Billing begins on the first of every month and runs through the end of the month. Billing will always be in arrears. For example, University IT (UIT) will bill April's usage in the May General Ledger accounting period (GL). The billing for April usage data won't be reflected in reports until the May GL period closes. You can visit the University IT Services Ordering & Billing page for more info on UIT billing.

What does "Project Task Award" refer to on the Add AI API Gateway Key form?

The Project Task Award, or PTA, is a method of labeling expenditure accounts in the university's financial systems. PTAs are used to categorize expenditures by funding source. You can learn more about PTAs in section 3.1.3 of the Stanford Admin Guide.

Do I always need to provide PTA billing information?

Yes. Pricing for these models changes regularly. Just because a model is available in public preview now, does not mean that it will always be that way. Providing PTA and budget information up front helps prevent interruptions to your service in the long run.

Please review the AI API Gateway Rates page for more information on pricing.

Why are the listed approvers for the PTA not correct?

The approvers for UIT billable services may be different than the PTA authority. These approvers will have the "UIT Billable Service Approver" role in Authority Manager. Please see the How to Request Authority for Ordering and Reporting page for more information.

How do I change which models I can access, the PTA associated with the key, or to change key owners?

To request changes to your existing keys, use the Request Assistance with the Stanford AI Playground form. Be sure to include the alias for your API account. Never include the API key in any form.

What is the difference between input tokens and output tokens? Why are they different prices?

When you use an AI model through the AI API Gateway, your requests and responses are measured in tokens.

Input Tokens: These tokens represent the text you submit to the AI model as a prompt.

Output Tokens: These tokens represent the text generated by the AI model in response to your prompt.

Input tokens and output tokens are billed at separate rates by the vendors, with output tokens typically priced higher due to the additional computational resources required to generate AI content. For detailed pricing rates, please refer to the AI API Gateway Rates page.

What happens if I exceed the max budget I selected when requesting the API key?

If the maximum budget defined is exceeded, the system will automatically prevent further API calls for the rest of that billing period. You will receive an error beginning with, "max budget error - liltellm".

To request a temporary extension of the max budget, complete the Request Assistance with the Stanford AI Playground form. Be sure to include the alias for your API account. Never include the API key in any form.

To permanently change the maximum monthly budget, complete the Request Assistance with the Stanford AI Playground form. Be sure to include the alias for your API account. Never include the API key in any form.

Data security and privacy

Can I use the APIs for sensitive or High-Risk data?

Per Stanford's Information Security Office and the University Privacy Office, the use of any models through the AI API Gateway requires a fully completed and approved Data Risk Assessment (DRA) before it can be considered approved for High Risk Data, Personally Identifiable Information (PII), or Protected Health Information (PHI). The AI Playground is not currently approved for High Risk Data, PII, or PHI.

Is information shared with the APIs used by UIT or vendors to train new models?

No. UIT has agreements in place with vendors covering model training and data retention. No data is used for training models. All information shared with the AI API Gateway Playground stays within the Stanford environment. Additionally, UIT is not training models nor are we using data shared with the AI API Gateway or AI Playground.

Does UIT have access to the data I share via the API? Are conversations logged or stored?

For security reasons, UIT logs information sent to through the AI API Gateway. We do not monitor this information, but in rare circumstances, as a result of investigations, subpoenas, or lawsuits, the university may be required to review data stored in university systems or provide it to third parties. You can learn more about the appropriate use of Stanford compute systems and these situations in the Stanford Admin Guide.

Can I use AI to fully automate critical or sensitive tasks?

It's important to remember that AI tools can make mistakes. While these technologies are powerful for augmenting decision making, they should not be used as a total replacement for human judgment. There should always be a human in the loop.

Human Oversight is the first of the Guiding Principles outlined in the Report of the AI at Stanford Advisory Committee. AI systems can generate helpful outputs and perform rudimentary tasks, but these outputs should be carefully reviewed by a human before being acted upon or shared.

Please see the related FAQ items below for additional guidance on commercial use of AI generated materials, citation of AI generated content, and the potential copyright considerations of AI generated works.

What should I do if the API key is lost or stolen?

You should use the Request Assistance with the Stanford AI Playground form to notify UIT immediately if the API key has been lost or otherwise compromised.