[python] OpenAPI Plugin: How to specify a 'base payload value'

[w4-jake]

It's been fun playing with the OpenAPI Plugin code. Here is a situation I am not sure is covered yet.

Overview

Straightforward case

Say that I am using Semantic Kernel on top of OpenAI to orchestrate various HTTP function calls for a client company, 'C'. 'C' has an endpoint at the URL http://client-company.com/remaining-vacation-days that accepts POST requests with a payload like below:

curl -X POST http://client-company.com/remaining-vacation-days \
  --data '{ "authCode": "SOME_CODE", "empno": "MY_EMPLOYEE_NUMBER" }' \
  -H "Content-Type: application/json"

The response is a simple JSON object, for example { "result": "25" } to indicate that the employee with MY_EMPLOYEE_NUMBER has twenty five remaining vacation days.

So far, this is no issue to handle with the OpenAPI plugin. I can define a YAML file as below:

openapi: 3.1.0
info:
  title: Employee API
  version: v0.1
servers:
  - url: http://client-company.com
paths:
  "/remaining-vacation-days":
    post:
      operationId: FindRemainingVacationDays
      description: Returns number of vacation days an employee has left
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                authCode:
                  type: string
                empno:
                  type: string
      responses:
        "200":
          description: {OK response description}
          content:
            application/json:
              schema:
                type: object
                properties:
                  result:
                    type: integer

I can use OpenAPI plugin code as below to generate a Kernel Plugin....

employee_plugin = KernelPlugin.from_openapi(
    plugin_name="Employee",
    openapi_document_path="./path/to/employee.yaml"
)

And ultimately Semantic Kernel communicates this plugin's function to OpenAI as follows...

{
    ... <omit>
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "Employee-FindRemainingVacationDays",
                "description": "Returns number of vacation days an employee has left",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "authCode": {
                            "type": "string",
                        },
                        "empno": {
                            "type": "string",
                        }
                    },
                    "required": []
                }
            }
        },
        ... <omit>

such that if an end user asks "How many vacation days do I have left?", the Kernel will prompt the user to input their authorization code and employee number in order to make the HTTP request.

Problem case

Say that for the purposes of the product it wants me to build, 'C' wants me to use the same authCode payload value for all requests. In this case, authCode should not be treated as an argument to the Kernel function. The function should behave equivalently to a manually-defined function with the signature below:

FIXED_AUTH_CODE = ... # Provided by 'C'

@kernel_function(
        name="find_remaining_vacation_days",
        description="Returns number of vacation days an employee has left",
    )
    def find_emp_vcatn_cnt(self, empno: Annotated[str, "some description"):
      BASE_PAYLOAD = { "authCode": FIXED_AUTH_CODE }
      request = request.post("http://client-company.com/remaining-vacation-days",
                                              json=BASE_PAYLOAD | { "empno": empno })

      # omit post process and return

Crucially,

1. authCode must not be passed inside thetools[i].function.parameters to OpenAI. OpenAI should ideally not know about authCode at all.
2. When this function is called, the FIXED_AUTH_CODE provided by 'C' must be included in the payload of the POST request.

In the scope of this thread, I will refer to payload values like authCode as "base payload values". They are fixed (hard-coded) in my server and are merged dynamically with payload values provided by OpenAI as parameters of the Plugin Function to create a complete payload for the HTTP request.

Solutions?

Unfortunately, I haven't found a way yet to use KernelPlugin.from_openapi(...) in a way that satisfies the requirements of this problem case. KernelPlugin.from_openapi(...) uses the OpenAPI protocol, which to my understanding was never designed to express "base payload values" since these values are unknown to the client using a given endpoint.

default?

Within the OpenAPI spec, you can mark a property with a default value like below:

      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                authCode:
                  type: string
                  default: FIXED_AUTH_CODE
                empno:
                  type: string

And using KernelPlugin.from_openapi(...) on a yaml file with this setting, I confirmed that this is also sent to OpenAI like so:

{
    ... <omit>
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "Employee-FindRemainingVacationDays",
                "description": "Returns number of vacation days an employee has left",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "authCode": {
                            "type": "string",
                            "default": FIXED_AUTH_CODE
                        },
                        "empno": {
                            "type": "string",
                        }
                    },
                    "required": []
                }
            }
        },
        ... <omit>

But unfortunately, this doesn't solve my issue. OpenAI sees authCode as an argument that needs to be filled in from somewhere, and in general asks the end user with a message like "Before I can check your vacation days, I need to know your authorization code. Can you provide me one? It might look something like {FIXED_AUTH_CODE}"

enum?

Using an enum property with a single value (stackoverflow) seems to be another way to use OpenAPI syntax to express a "base payload value":

      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                authCode:
                  type: string
                  enum: ["FIXED_AUTH_CODE"]
                empno:
                  type: string

but alas, authCode is similarly sent to OpenAI...

{
    ... <omit>
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "Employee-FindRemainingVacationDays",
                "description": "Returns number of vacation days an employee has left",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "authCode": {
                            "type": "string",
                            "enum": ["FIXED_AUTH_CODE"]
                        },
                        "empno": {
                            "type": "string",
                        }
                    },
                    "required": []
                }
            }
        },
        ... <omit>

and treated as an argument that the end user must provide, so the issue remains unsolved.

const?

It looks like starting in OpenAPI 3.1, a const keyword is being introduced that could most intuitively express a "base payload value". (stackoverflow). But I do not believe that this keyword is being handled in openapi_parser.py at the moment.

Discussion Points

• It seems that if authCode were not a fixed payload value sent as the body of a POST request, but were instead a fixed header, there are workarounds available. I could either create a custom http_client that will always use the fixed headers I want, or perhaps use the auth_callback option, though I have not looked too deeply into it yet. (Is there a way to specify headers for OpenAPI Plugins without explicitly doing so in the query? #5662)
  • But say that I am not in a position to negotiate with 'C' to accept authCode as a header, so it must go as a payload value (body of the POST request).
  • For that matter, we could imagine a similar situation in which 'C' demands a fixed value as a query to a given GET request, so that there is such a thing as a "base query value".
• Ideally, I would love to be able to continue using KernelPlugin.from_openapi(...) in a situation like this. It seems that the two directions to go to support it, if there is no way to do so already, would be:
  • (A) to introduce support for this feature through the const keyword or some other existing OpenAPI syntax, such that that syntax is interpreted as a 'base payload value' (or 'base query value'?), or
  • (B) to avoid expressing this situation using OpenAPI syntax and instead allow for a developer to customize 'base payload values' (and 'base query values'?) in the OpenAPIFunctionExecutionParameters.

Of course, if I have overlooked an easier solution to this problem using current code, I would be thrilled to be shown what to do! I am still looking into what some of the options like enable_dynamic_payload do, so maybe I will be pleasantly surprised.

Thank you for any guidance!

[moonbox3]

Hi @w4-jake, thanks for the detailed question/post/discussion. This is wonderful.

After an initial read, one should be able to adapt this similar to the following OpenAPI Azure Key Vault sample/yaml:

• Sample code: https://github.com/microsoft/semantic-kernel/blob/main/python/samples/concepts/plugins/openai_plugin_azure_key_vault.py
• Sample OpenAPI spec: https://github.com/microsoft/semantic-kernel/blob/main/python/samples/concepts/resources/open_ai_plugins/akv-openapi.yaml

You can see in the following code that the kernel arguments contain some default values to create a secret:

https://github.com/microsoft/semantic-kernel/blob/821968a2631e4c3ce41fb953bbad1e52ac8a9eed/python/samples/concepts/plugins/openai_plugin_azure_key_vault.py#L142C1-L154C50

async def add_secret_to_key_vault(kernel: Kernel, plugin: KernelPlugin):
    """Adds a secret to the Azure Key Vault."""
    arguments = KernelArguments()
    arguments["secret_name"] = "Foo"  # nosec
    arguments["api_version"] = "7.0"
    arguments["value"] = "Bar"
    arguments["enabled"] = True
    result = await kernel.invoke(
        function=plugin["SetSecret"],
        arguments=arguments,
    )

which corresponds to this portion of the OpenAPI spec:

put:
      summary: Create or update secret value
      description: "Sets a secret in a specified key vault. This operation adds a secret to the Azure Key Vault. If the named secret already exists, Azure Key Vault creates a new version of that secret. This operation requires the secrets/set permission. For details, see: https://learn.microsoft.com/en-us/rest/api/keyvault/secrets/set-secret/set-secret."
      operationId: SetSecret
      parameters:
        - name: secret-name
          in: path
          required: true
          schema:
            type: string
        - name: api-version
          in: query
          required: true
          schema:
            type: string
            default: "7.0"
          x-ms-visibility: internal
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                attributes:
                  type: object
                  properties:
                    enabled:
                      type: boolean
                      description: Determines whether the object is enabled.
                value:
                  type: string
                  description: The value of the secret.
              required:
                - value

The AKV spec structure allows one to set up the request body to include the enabled bool and the value string. This would be similar to your authCode key value pair that you want to include in the request body. The model shouldn't need to know about this information because it's only related to the API call, and not part of a tool call.

Could you have a look at this, please, and see if we're on track to help with your use-case? Otherwise, I can keep digging. Thanks!

[w4-jake]

Hi @moonbox3, thanks for the response. I'm happy to know that this could be a helpful discussion!

I would need to do some prep in order to directly run the openai_plugin_azure_key_vault.py sample that you shared — looks like I'd need to set up credentials for an Azure Key Vault service? — but taking a look at the code, I see what you mean that it is possible to manually call the generated function plugin["SetSecret"] by setting specific fields within the arguments passed to kernel.invoke. This would be a way to ensure that authCode is always used with a set value like I described.

But I'm not sure yet about your description that

The model shouldn't need to know about this information because it's only related to the API call, and not part of a tool call.

Focusing just on the value property that is part of the PUT request, since this is most similar to my authCode, the path string to OpenAPI information about this property...

semantic-kernel/python/samples/concepts/resources/open_ai_plugins/akv-openapi.yaml

Lines 89 to 91 in 2200e00

	value:
	type: string
	description: The value of the secret.

looks like it will be passed inside openai_spec when adding a plugin here:

semantic-kernel/python/samples/concepts/plugins/openai_plugin_azure_key_vault.py

Lines 248 to 257 in 2200e00

	await kernel.add_plugin_from_openai(
	plugin_name="AzureKeyVaultPlugin",
	plugin_str=openai_spec,
	execution_parameters=OpenAIFunctionExecutionParameters(
	http_client=http_client,
	auth_callback=authentication_provider.authenticate_request,
	server_url_override=str(endpoint),
	enable_dynamic_payload=True,
	),
	)

So ultimately the path will be passed to create_functions_from_openapi here, where the information about the value property is parsed and used to generate a kernel function:

semantic-kernel/python/semantic_kernel/connectors/openapi_plugin/openapi_manager.py

Lines 33 to 37 in 2200e00

	def create_functions_from_openapi(
	plugin_name: str,
	openapi_document_path: str,
	execution_settings: "OpenAIFunctionExecutionParameters | OpenAPIFunctionExecutionParameters | None" = None,
	) -> list[KernelFunctionFromMethod]:

Despite this, when this kernel function is mapped to "tools" metadata to be passed to the LLM (in my case, using OpenAI), will the value property not be passed as a parameter in some form like below?

!!! speculation, not actually generated!

{
    ... <omit>
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "AzureKeyVaultPlugin-SetSecret",
                "description": "...",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "value": {
                            "type": "string",
        ... <omit>

Without running the sample code to check empirically it is hard to say for sure, but it seems that when a kernel with this plugin is used, the LLM will still be told that a string argument called "value" is needed to call the tool, and thus the LLM will suggest asking directly for the end user to provide this value.

But maybe this is not the case here? Perhaps if I get more time next week, I could try to set up everything I need to run the sample code directly.

---

For a little more background about why I am asking about this, I am planning to support not just the one client 'C' that I described, but multiple clients all with different requirements for their API calls. So I want to be able to take in any given client company's OpenAPI YAML manifest and build a plugin out of it using KernelPlugin.from_openai(...) or KernelPlugin.from_openapi(...), and ideally I would want to be able to define 'base payload values', 'base headers', and 'base query values' within the syntax of the OpenAPI YAML manifest.

But since (to my knowledge) OpenAPI was never designed to express these kinds of features, I may need to write my own parsing logic that treats some OpenAPI syntax, like the const keyword, in this manner. But I'd need to do a lot of "wheel reinventing" in order to do this, so I'm trying to think of a better solution.