HuggingFacePromptExecutionSettings Class

Show details of the generation. Including usage.

(Optional: True). Bool. Whether or not to use sampling, use greedy decoding otherwise.

Extra properties that may be included in the serialized execution settings.

Gets or sets the behavior defining the way functions are chosen by LLM and how they are invoked by AI connectors.

Gets a value that indicates whether the PromptExecutionSettings are currently modifiable.

Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message.

Int (0-250). The amount of new tokens to be generated, this does not include the input length it is a estimate of the size of generated text you want. Each new tokens slows down the request, so look for balance between response times and length of text generated.

(Default: None). Float (0-120.0). The amount of time in seconds that the query should take maximum. Network can cause some overhead so it will be a soft limit. Use that in combination with max_new_tokens for best results.

The maximum number of tokens to generate in the completion.

Model identifier. This identifies the AI model these settings are configured for e.g., gpt-4, gpt-3.5-turbo

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics

(Default: None). Float (0.0-100.0). The more a token is used within generation the more it is penalized to not be picked in successive generation passes.

(Default: 1). Integer. The number of proposition you want to be returned.

(Default: True). Bool. If set to False, the return results will not contain the original query making it easier for prompting.

The seed to use for generating a similar output.

Service identifier. This identifies the service these settings are configured for e.g., azure_openai_eastus, openai, ollama, huggingface, etc.

Up to 4 sequences where the API will stop generating further tokens.

(Default: 1.0). Float (0.0-100.0). The temperature of the sampling operation. 1 means regular sampling, 0 means always take the highest score, 100.0 is getting closer to uniform probability.

(Default: None). Integer to define the top tokens considered within the sample operation to create new text.

An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

(Default: None). Float to define the tokens that are within the sample operation of text generation. Add tokens in the sample for more probable to least probable until the sum of the probabilities is greater than top_p.

(Default: true). Boolean. There is a cache layer on the inference API to speedup requests we have already seen. Most models can use those results as is as models are deterministic (meaning the results will be the same anyway). However if you use a non deterministic model, you can set this parameter to prevent the caching mechanism from being used resulting in a real new query.

(Default: false) Boolean. If the model is not ready, wait for it instead of receiving 503. It limits the number of requests required to get your inference done. It is advised to only set this flag to true after receiving a 503 error as it will limit hanging in your application to known places.

Creates a new PromptExecutionSettings object that is a copy of the current instance.

Makes the current PromptExecutionSettings unmodifiable and sets its IsFrozen property to true.

Gets the specialization for the HuggingFace execution settings.

Throws an InvalidOperationException if the PromptExecutionSettings are frozen.