Skip to content
GitHub

API Reference

Classes

Section titled “Classes”

Result of running evaluation on multiple samples.

Contains aggregated results and per-sample details.

Configuration for the evaluation subsystem.

Attributes: enabled: Enable the AI evaluation subsystem. default_threshold: Default score threshold for passing evaluations. embedding_model: Model to use for embedding-based evaluations. include_metadata: Whether to include metadata in run reports. max_samples: Maximum number of samples per evaluation run. max_retries: Maximum retries for failed evaluations. timeout_seconds: Timeout for evaluation execution in seconds.

Example

config = EvaluationConfig(
    default_threshold=0.9,
    embedding_model="text-embedding-3-large"
)
config = EvaluationConfig(
    default_threshold=0.9,
    embedding_model="text-embedding-3-large"
)

A collection of evaluation samples.

Attributes: name: Name of the dataset. samples: List of evaluation samples. metadata: Additional dataset metadata.

Evaluation module for Lexigram applications.

Provides evaluator and harness support for AI model evaluation.

Usage

from lexigram.ai.evaluation import EvaluationModule
from lexigram.ai.evaluation.config import EvaluationConfig


@module(
    imports=[EvaluationModule.configure(EvaluationConfig(...))]
)
class AppModule(Module):
    pass
from lexigram.ai.evaluation import EvaluationModule
from lexigram.ai.evaluation.config import EvaluationConfig


@module(
    imports=[EvaluationModule.configure(EvaluationConfig(...))]
)
class AppModule(Module):
    pass
configure
def configure(
    cls,
    config: EvaluationConfig | None = None
) -> DynamicModule

Create an EvaluationModule with explicit configuration.

Parameters
ParameterTypeDescription
`config`EvaluationConfig | NoneEvaluationConfig or ``None`` for defaults.
Returns
TypeDescription
DynamicModuleA DynamicModule descriptor.
stub
def stub(
    cls,
    config: EvaluationConfig | None = None
) -> DynamicModule

Create an EvaluationModule suitable for unit and integration testing.

Uses in-memory or no-op evaluator implementations with minimal side effects.

Parameters
ParameterTypeDescription
`config`EvaluationConfig | NoneOptional EvaluationConfig override. Uses safe test defaults when ``None``.
Returns
TypeDescription
DynamicModuleA DynamicModule descriptor.

Registers evaluation services with the DI container.
__init__
def __init__(config: EvaluationConfig | None = None) -> None
register
async def register(container: ContainerRegistrarProtocol) -> None
boot
async def boot(container: BootContainerProtocol) -> None
shutdown
async def shutdown() -> None
health_check
async def health_check(timeout: float = 5.0) -> HealthCheckResult

Result of an evaluation run on a single sample.

Attributes: score: The evaluation score (0.0 to 1.0). score_type: The type of scoring method used. feedback: Human-readable feedback about the evaluation. metrics: Additional metrics computed during evaluation.

Context for a single evaluation run.

Holds the dataset, evaluator, and configuration for an evaluation run.

A single sample in an evaluation dataset.

Attributes: id: Unique identifier for this sample. input: The input prompt or query. reference: The expected reference output. metadata: Additional metadata for this sample.

Report from running an evaluator on a dataset.

Attributes: dataset_name: Name of the evaluated dataset. evaluator_name: Name of the evaluator used. total_samples: Total number of samples evaluated. passed_samples: Number of samples that passed the evaluation. average_score: Average score across all samples. results: Individual sample results. metadata: Additional report metadata.

Exceptions

Section titled “Exceptions”

Raised when there's an error with the evaluation dataset.

Raised when evaluation configuration is invalid.

Raised when a requested evaluator cannot be found.

Raised when the evaluation harness encounters an error.