Preview Feature: Agent Tuning

You can create, evaluate, and automatically tune multiple versions of an agent, although only one is active at a time. This allows you to test and compare outcomes, choosing the best-performing version to activate. You can also use it for version control: keep the old version as a backup.

For each tuning version of an agent, you can:

View and manually edit custom instructions
View and manually refine column descriptions
Evaluate the version to measure how well the agent is performing
- The answer to each question in the selected question set is evaluated as passing or failing (status of expected result or flagged as unexpected).
- Evaluation information for each question includes the agent's answer and interpretation, the expected interpretation and SQL, and performance statistics.
- You can then choose to auto tune the version
Auto tune the version, that is, automatically adjust and optimize the agent
Download the tuning file to reuse and share the information among agents

This is available as a preview feature in Strategy (October 2026).

Preview features are early versions of features and are not to be used in a production environment as the core behavior remain subject to change between preview and GA. By selecting to expose preview features, you can access these features and use them as you would any other functionality. The official versions of preview features are included in subsequent releases.

Prerequisites

This feature is available as a preview feature. Enable the Agent Tuning Preview Feature.

To access the AI Admin console, you need the AI Administrator privilege.
To access the Agent Optimization section in the AI Admin console, you need the Agent Administrator privilege.

Enable the Agent Tuning Preview Feature

In Library Server, add these lines to the config.Override.properties file:
- services.ai.v2.aiAdminEnabled=true
- services.ai.v2.agentAdminEnabled=true
- features.sidebar.adminmodel.entrypoint=true

Access the Agent Tuning window

In the Library sidebar, click AI Admin.
In the AI Admin sidebar, click Agent Optimization.
Click the Agent Tuning Versions tab.

Manage the Agent

Access the Agent Tuning window.
All agents display information such as status (Active vs. Inactive) and owner. To hide and display columns, click Columns on the right.
To upload a tuning file and create another version for the agent:
1. Click More for the agent and select Upload New Version.
2. Navigate to the folder where the tuning file is saved, select it, and click Open.
3. In the Save As window, type the Version Name and click Save.
To view the versions for an agent, click the agent in the grid. Versions display sorted by the modification date. Only one version is active for each agent.
To interact with the agent, click More for the agent and select Open Agent.
To de-activate the agent so that users cannot access it, click More for the agent and select De-Activate.
- You can re-activate an inactive agent. Click More for the agent and select Activate.

Manage Agent Versions

Access the Agent Tuning window.
To view the versions for an agent, click the agent in the grid. Versions display sorted by the modification date. Only one version is active for each agent.
To apply a version to the agent, right-click a Not In Use version and select Publish.
- If you previously certified the agent, the certification status does not change.
- If another user certified the agent, you are prompted to certify it.
To duplicate a version, right-click it and select Duplicate. To rename the copy, right-click it, select Rename, and type the new name.
To permanently delete a version, right-click it and select Delete. Click Delete to confirm.
To evaluate a version, right-click it and select Evaluate.
To auto tune a version, right-click it and select Auto Tune.
To view the version's custom instructions and column descriptions, click the version. The Version Preview window opens.

Download the Tuning File

At any point, you can download the current settings as a tuning file. Use this to save a snapshot of the settings or share them with other agents.
Click Download Tuning File.

View and Edit Custom Instructions
Click Custom Instructions.
Toggle on or off Enable Custom Instructions.
When custom instructions are enabled, you can:
- Describe your business background and any special requirements the agent should consider for the data. Hover your cursor over the Information icon to view examples.
  The character limit is 1500 characters. This field does not display for universal agents. The requirements are determined separately in each agent definition.
- Specify the format of the responses the agent should use, such as the structure, tone, and sentiment for responses. Default instructions are provided, but you can remove, edit, and customize the format instructions. For example, you can instruct the agent to use bullet points to organize information, use an informal tone, and provide positive feedback at a certain threshold. Hover your cursor over the Information icon to view examples. For an example, see Best Practices: Working with Agents.
  The character limit is 1500 characters.
- Specify the types of questions that should be rejected at the agent level, that is, that the agent does not answer. Examples include "Do not discuss topics related to profit" and "Do not provide information about social security numbers". For an example, see Best Practices: Working with Agents.
  The character limit is 2500 characters.
View and Refine Column Descriptions
Click Column Descriptions.
If you have multiple datasets or agents, you can focus on a specific dataset or agent by selecting it from the drop-down list.
Edit the dataset and column descriptions. For a universal agent, edit the agent descriptions.

It is recommended that you do not delete any of the automatic description but instead add to them. For information on what is automatically generated and how to enhance it, see Refine Automatically-Generated Descriptions.
If you make any changes to the custom instructions or column descriptions, click Save.
- If the version is in use by the agent, confirm that you want these changes to apply to the active agent.
You can evaluate this version. Click More at the top of the window and select Evaluate.
You can auto tune this version. Click More at the top of the window and select Auto Tune.

Download the Tuning File
To create a file of the custom instructions and column descriptions, click Download Tuning File.

Apply The Version to the Agent

If you do not want to apply the version, close X the Version Preview window.
Click Publish.
- If you previously certified the agent, the certification status does not change.
- If another user certified the agent, you are prompted to certify it.

Evaluate an Agent Version

Access the Agent Tuning window.
To view the versions for an agent, click the agent in the grid. Versions display sorted by the modification date.
Right-click a version and select Evaluate.
In the Evaluate Agent window, select the question set to evaluate against by doing one of the following:
- Select a existing question set from the drop-down list.
- Upload a new question set from your local files by clicking the Upload icon .
Enter the number of Run Cycles, which is the number of runs for each question in the set.
Select whether to Execute the Query against the warehouse data.
Click Evaluate.
The Job Monitoring tab displays, listing all currently running and completed jobs. Use the tab to monitor the job, then review and follow up on the results.

Auto Tune an Agent Version

Access the Agent Tuning window.
To view the versions for an agent, click the agent in the grid. Versions display sorted by the modification date.
Right-click a version and select Auto Tune.
In the Auto Tune Agent window, select the question set by doing one of the following:
- Select a existing question set from the drop-down list.
- Upload a new question set from your local files by clicking the Upload icon .
Enter the number of Run Cycles, which is the number of runs for each question in the set.
Click Auto Tune.
The Job Monitoring tab displays, listing all currently running and completed jobs. Use the tab to monitor the job, then review and follow up on the results.

Monitor Jobs

In the Library sidebar, click AI Admin.
In the AI Admin sidebar, click Agent Optimization.
Click the Job Monitoring tab. All currently running and completed jobs are listed, with the question set they used and their status. Use this tab to monitor the job status, then review and follow up on the results.

Once a job is complete, its status is updated. Use the table below to understand each status and what action to take.

Status	Description	Action
Finished	The job has completed with no errors.	For an evaluation job, click View to see the test results and make tuning changes. For an auto tuning job, click View to see the auto tuning results and accept or discard them.
Viewed	You have viewed the job's test results without making changes to the custom instructions or column descriptions.	For an evaluation job, click Open to view the test results again and make tuning changes. For an auto tuning job, click Open to see the auto tuning results again and accept or discard them.
Incomplete Baseline	The question set was found to be incomplete during auto tuning.	Do either of the following: Click View to define the missing answers and then run the auto tuning again. Click Re-Run to run the auto tuning again.
Expired	The test results are invalid because you changed the custom instructions or column descriptions.	For an evaluation job, there is no action because the evaluation is outdated. For an auto tuning job, click Open to view the now outdated test results again and save them as a new version.

View Test Results for an Evaluation

When you click View or Open for an evaluation job, the Test Results window lists each question in the question set, with its status of either Flagged or Expected, and performance statistics. An overall accuracy rate for the entire question set is also provided.
Click Custom Instruction or Column Descriptions to view the information. You can enable or disable Show Diffs Since Last Save. Differences are highlighted in purple.
Inspect the response from an individual question by clicking the question. The agent's answer, interpretation, and SQL explanation are provided.
You can view the Expected Interpretation or the Expected SQL.
If you decide that the agent's response is correct but the evaluation flagged the response, change the answer status. Click Flagged and select Expected. The tuning AI learns that the question can be answered in this way. The Cache Manager updates.
If you decide that the agent's response is not correct but the evaluation did not flag the response, change the answer status. Click Expected and select Flagged. The tuning AI learns that the answer is not correct and updates its judgment logic. The Cache Manager removes the answer.
If you change the answer status, you can edit the expected answer. Click Edit, make your changes, and click Save.
To review a different question, click the Select Question icon .
To return to the full list of questions, click the arrow icon < next to Test Results.
To run auto tuning, click the Auto Tune icon on the toolbar. For information, see View Test Results for Auto Tuning.
To save changes, click the Save icon on the right of the toolbar.
To apply this version to the agent,click Publish.
Click X to close the Test Results window and return to the Job Monitoring tab.
View Test Results for Auto Tuning
When you click View or Open for an auto tuning job, the Test Results window lists each question in the question set, with its current accuracy and the accuracy after the proposed tuning is accepted.
Click Custom Instruction or Column Descriptions to view the information. You can enable or disable Show Diffs Since Last Save. Differences are highlighted in purple.
To discard the auto tuning suggestions, click Discard, then close X the Test Results window and return to the Job Monitoring tab.
To instead accept the auto tuning suggestions, click Accept Tuning. Choose one of the following options:
- To apply the changes to the current version, overwriting the existing information, select Apply to Current Version.
- To save the changes as a new version, select Save as New Version. Type the Version Name and click Save.
Click X to close the Test Results window and return to the Job Monitoring tab.
Define Missing Answers for an Incomplete Baseline Job
When you click View for an incomplete baseline job, each question with a missing expected answer displays, listing the question and the agent's answer and interpretation.
Do one of the following:
- To accept the answer as is, click Accept Agent Answer as Expected.
- To instead add your answer, click Add Expected Answer. Type the text of the answer in the panel, then click Save.
To scroll through the questions, use the arrow keys at the bottom of the Incomplete Baseline window.
Once all questions have answers, click Run Evaluation. You can then View Test Results for Auto Tuning.
View Outdated Test Results for Auto Tuning
When you click Open for an expired auto tuning job, the Test Results window lists each question in the question set, with its current accuracy and the accuracy after the proposed tuning is accepted.
Save the auto tuned results as a new version.
Click X to close the Test Results window and return to the Job Monitoring tab.

Manage the Question Sets

Question sets are used in evaluation and auto tuning.

In the Library sidebar, click AI Admin.
In the AI Admin sidebar, click Agent Optimization.
Click the Question Set tab to view the existing question sets.
To download a template to use to create a new question set, click Question Set Template in the toolbar. A JSON file is downloaded.
To create a new question set:
1. Click New Question Set in the toolbar.
2. Do one of the following:
  - Create a new question by clicking Create and Design a Completely New Question.
  - Select an existing question from the list.
3. Click Next Step.
4. Type the Answer Instructions for a new question or update them for an existing question.
5. Optionally, add SQL in the Expected SQL Reference box for a new question or update it for an existing question.
6. To add another question, click Back to List and repeat the steps.
7. To complete the question set, click Save & Finalize.
To manage an existing question set, right-click it and select one of the following options:
- Edit
- Rename
- Download
- Duplicate
- Delete