AI API Gateway FAQs

Below is a list of expected questions and answers for the AI API Gateway.

Basics

What is the AI API Gateway?

The AI API Gateway is a centralized service providing direct API access to the advanced AI models featured in the Stanford AI Playground. The AI API Gateway uses the opensource LiteLLM platform to help enable teams to develop AI tools and integrate AI capabilities into your applications.

What models are accessible through the AI API Gateway?

The AI API Gateway has access to all of the major Large Language Models available in the AI Playground. View the AI API Gateway Rates page for a complete list of the currently available models.

What is an API endpoint?

API endpoints are specific URLs that support specific types of API requests and return responses. Endpoints exist on a server so that clients can use those addresses to access resources or functionality provided by an API.

Getting started

How do I request an API key?

You can use the Add AI API Gateway Key form to request the creation of a new API key.

How do I get started with my API key?

Once you receive an API key, follow the guidance in the AI Playground Access and Usage help page. 

What endpoints are available in the AI API Gateway?

For a complete list of supported endpoints, please view the AI Playground Access and Usage help page. 

Not all models support usage of the same endpoints. Make sure your API key has access to an appropriate model when trying to call the respective endpoint.

How do I share or provide API access to other people on my team/cohort?

Once your API key request ticket has been processed, the AI Playground team will provide your key via a secure email. The key is issued to the individual requester and should be handled securely. If others on your team need access, you should coordinate with them directly to responsibly share the API key provided or they may submit their own request for a separate key.

You are free to share this key with anyone you deem should have it, but please note that anyone with your API key would be able to accrue charges against the PTA you provided. If you are concerned with other people using your PTA, it may be best for them to request their own API key.

How can I update the owner, available AI models, budget, etc. that is associated with my API key?

You can use the Change or Cancel AI API Gateway Key Service form to request changes to an existing API key.

When filling out the API Key request form, how accurate do my estimated daily API requests need to be?

Try to be reasonably accurate to help with system planning and bandwidth allocation. However, we understand these numbers are often approximations and may not reflect exact daily usage.

I requested a key from the ticket, but haven’t received the key yet. How long does it take to generate an API key?

It generally takes two to five days to complete all the setup for the request, assuming that the PTA owner has approved. 

Before opening a new ticket:

• If your ticket has not been completed, please check that the PTA approver you selected has approved the request before opening a ticket.
• If the PTA owner has approved, and you have not received the API key within five business days, then please add a comment to your original ticket. 

If the ticket is closed and still you have not received your key, then please open a ticket to inquire about the issue.

Using the AI API Gateway

I receive an error reading, "API Key not allowed to access model." How do I fix this?

It sounds as if you either have a typo in the model name you are using or you are trying to access a model for which you did not request access.

Double check the name of the model you are using with what is listed in the KB article, "Stanford AI API Gateway Access and Usage".

If it is spelled correctly, please open a ticket to request access to the new model be associated with the API key you are using. 

Does the AI API Gateway support batch processing?

Batch processing is not currently available through the AI API Gateway. At this time, all requests must be made individually. However, batch request functionality is on our roadmap for future development, and updates will be shared once it becomes available.

The GPT 4omni model name is shown as "gpt-4.omini" in the AI API Gateway. Is that a typo?

This is not a typo. While the "o" in the GPT 4o model series does stand for "Omni", this model is the GPT 4o mini model. You can view the correct names for each model in the KB article, "Stanford AI API Gateway Access and Usage".

Cost tracking, budgets, and expense

How much does it cost to use the AI API Gateway?

The AI API Gateway is a chargeback service based on usage. Rates are determined by our vendors and the negotiated Stanford discount. Please note that the vendors change these prices often. 

For detailed pricing rates, please refer to the AI API Gateway Rates page. During your service setup, you will be asked to indicate a max budget, which will prevent API calls beyond that budget. 

How does billing work?

Billing begins on the first of every month and runs through the end of the month. Billing will always be in arrears. For example, University IT (UIT) will bill April's usage in the May General Ledger accounting period (GL). The billing for April usage data won't be reflected in reports until the May GL period closes. 

For more information specific to billing, you can visit the University IT Services Ordering & Billing page.

How can I see the current spending on my API key?

Real time reporting of API key usage and spending is not yet available, though it is on the service roadmap. At present, billing begins on the first day of each month and runs through the end of the month. Charges are always applied in arrears, meaning usage for a given month will appear on the following month’s billing cycle.

To view charges for the past month, you will need to log into the UIT Billing Dashboard. This tool provides staff an interactive visualization tool to view the charges for their department. You can learn more about the dashboard, as well as how to access it, on the How to View Billing Details Using UIT Billing Dashboard page. If you cannot see the dashboard, you will need to have the "Request Reporting" role granted in Authority Manager.  Each budget unit has at least 2 people in their org who can grant them that role.  Additional details can be found on at How to Request Authority for Ordering and Reporting.

What does the "maximum monthly budget" field on the API Key request form represent?

The monthly budget serves as a cost control mechanism. Once your usage reaches this budget, the API key is temporarily disabled. You can either wait until the start of the next month or open a support ticket to reenable the key.

Will I be charged the full monthly budget even if usage is lower?

No. You are only charged for your actual usage. The budget field acts purely as a cost control mechanism to prevent unexpected expenses.

What happens if my monthly budget is exceeded?

If you exceed the monthly budget, your API key access is automatically disabled until the end of the month. If that happens, you can either wait until the start of the next month or open a support ticket to reenable the key.

Is there a recommended "normal" monthly budget for API usage?

We do not currently have a standardized model or recommended budget for API expenses, as use cases vary significantly across campus and are based on application specifics.

Is there an estimate on typical daily usage for a simple chatbot?

Usage can vary widely based on implementation details, frequency of interactions, and complexity of user queries. Currently, we do not have a standard model for predicting usage.

What does "Project Task Award" refer to on the Add AI API Gateway Key form?

The Project Task Award, or PTA, is a method of labeling expenditure accounts in the university's financial systems. PTAs are used to categorize expenditures by funding source. You can learn more about PTAs in section 3.1.3 of the Stanford Admin Guide.

Do I always need to provide PTA billing information?

Yes. Pricing for these models changes regularly. Just because a model is available in public preview now, does not mean that it will always be that way. Providing PTA and budget information up front helps prevent interruptions to your service in the long run. 

Please review the AI API Gateway Rates page for more information on pricing.

Why are the listed approvers for the PTA not correct?

They are correct. What you are seeing is expected behavior. The approver list for UIT billable services may be different than those listed in the PTA authority or other systems. 

The UIT Billable system prioritizes the lower ORG level first. A user with access to a top level org code, would need to request the UIT Billable Service Approver (University) privilege under the specific ORG code to be included in the approvers list. The approvers you see listed when filling out the API request form, will have the "UIT Billable Service Approver" role in Authority Manager for the lower ORG code. 

Please see the How to Request Authority for Ordering and Reporting page for more information. 

Why do I get an error saying "You do not have authority over the service account entered" when trying to change or cancel the API key?

The API Key is associated with the user who submitted the original request. The person who requested the API key will need to be the person who requests changes to the key.

If a department was selected instead of an individual at the time the key was created, then only users with ORBIT authority will be able to submit changes.

If there is a blocker, the original requestor can submit a ticket via the Request Assistance with the Stanford AI Playground form to change or add approved users.

I requested the API key, but am not able to request changes when using the Change or Cancel API Key form?

If the original requestor of the API key selected that the key was for a team or department at the time the key was created, only users with ORBIT authority in that department will be able to submit changes. You will need your manager to grant you ORBIT authority via Authority Manager or have someone on your team who has ORBIT authority request the API key change.

If you need to change this, the original requestor can submit a ticket via the Request Assistance with the Stanford AI Playground form to request an override to the approved users.

What is the difference between input tokens and output tokens? Why are they different prices?

When you use an AI model through the AI API Gateway, your requests and responses are measured in tokens.

Input Tokens: These tokens represent the text you submit to the AI model as a prompt. 

Output Tokens: These tokens represent the text generated by the AI model in response to your prompt. 

Input tokens and output tokens are billed at separate rates by the vendors, with output tokens typically priced higher due to the additional computational resources required to generate AI content. For detailed pricing rates, please refer to the AI API Gateway Rates page.

When does my monthly AI API Gateway budget reset?

The AI API Gateway budget resets every 30 days based on the date the key was created.

For example, if your account was created on May 5, your budget will reset on the 4th of June.

What happens if I exceed the max budget I selected when requesting the API key?

If the maximum budget defined is exceeded, the system will automatically prevent further API calls for the rest of that billing period. You will receive an error beginning with, "max budget error - liltellm". 

The AI API Gateway budget resets every 30 days based on the date the key was created.

To request a temporary extension of the max budget, complete the Request Assistance with the Stanford AI Playground form. Be sure to include the alias for your API account. Never include the API key in any form. 

To permanently change the maximum monthly budget, complete the Request Assistance with the Stanford AI Playground form. Be sure to include the alias for your API account. Never include the API key in any form.

Data security and privacy

Can I use the APIs for sensitive or High-Risk data?

Per Stanford's Information Security Office and the University Privacy Office, the use of any models through the AI API Gateway requires a fully completed and approved Data Risk Assessment (DRA) before it can be considered approved for High Risk Data, Personally Identifiable Information (PII), or Protected Health Information (PHI). The AI Playground is not currently approved for High Risk Data, PII, or PHI.

Is information shared with the APIs used by UIT or vendors to train new models?

No. UIT has agreements in place with vendors covering model training and data retention. No data is used for training models. All information shared with the AI API Gateway Playground stays within the Stanford environment. Additionally, UIT is not training models nor are we using data shared with the AI API Gateway or AI Playground.

Does UIT have access to the data I share via the API? Are conversations logged or stored?

For security reasons, UIT logs information sent to through the AI API Gateway. We do not monitor this information, but in rare circumstances, as a result of investigations, subpoenas, or lawsuits, the university may be required to review data stored in university systems or provide it to third parties. You can learn more about the appropriate use of Stanford compute systems and these situations in the Stanford Admin Guide.

Do the API keys provided by the AI API Gateway store or retain conversation data?

Yes. Per ISO requirements, UIT logs information sent to through the AI API Gateway. UIT does not monitor the conversations or uploads shared with the AI API Gateway. In rare circumstances, as a result of investigations, subpoenas, or lawsuits, the university may be required to review data stored in university systems or provide it to third parties. You can learn more about the appropriate use of Stanford compute systems and these situations in the Stanford Admin Guide.

Can I request that UIT refrain from logging data related to my API key?

No. Per ISO requirements, UIT logs information sent to through the AI API Gateway within the Stanford environment. UIT does not monitor this information, but in rare circumstances, as a result of investigations, subpoenas, or lawsuits, the university may be required to review data stored in university systems or provide it to third parties. You can learn more about the appropriate use of Stanford compute systems and these situations in the Stanford Admin Guide.

Can I use AI to fully automate critical or sensitive tasks?

It's important to remember that AI tools can make mistakes. While these technologies are powerful for augmenting decision making, they should not be used as a total replacement for human judgment. There should always be a human in the loop. 

Human Oversight is the first of the Guiding Principles outlined in the Report of the AI at Stanford Advisory Committee. AI systems can generate helpful outputs and perform rudimentary tasks, but these outputs should be carefully reviewed by a human before being acted upon or shared. 

Please see the related FAQ items below for additional guidance on commercial use of AI generated materials, citation of AI generated content, and the potential copyright considerations of AI generated works.

What should I do if the API key is lost or stolen?

You should use the Request Assistance with the Stanford AI Playground form to notify UIT immediately if the API key has been lost or otherwise compromised.