Integration Testing

Integration testing in KubeOps involves testing your operator against a real Kubernetes cluster. This ensures that your operator works correctly in an environment similar to production.

note

The integration testing tools shown in this documentation are currently internal to KubeOps and used in its own test suite. They are not yet available as a separate package. You can use these examples as a reference for implementing your own integration testing infrastructure.

CI/CD Setup

KubeOps (itself) uses GitHub Actions for CI/CD testing. Here's a typical setup:

name: .NET Testing

on:
  pull_request:
    branches:
      - "**"

concurrency:
  group: testing-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    name: Testing
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup .NET
        uses: actions/setup-dotnet@v3
        with:
          dotnet-version: 9.x

      - name: Create Kubernetes Cluster
        uses: helm/kind-action@v1.12.0

      - name: Execute Tests
        run: dotnet test --configuration ${{ runner.debug == '1' && 'Debug' || 'Release' }}

This setup:

1. Creates a Kubernetes cluster using kind
2. Builds and runs your tests against the cluster
3. Cleans up resources after tests complete

Test Infrastructure

The following examples show how KubeOps implements its own integration tests. You can use these patterns to implement your own integration testing infrastructure:

Test Collection and CRD Installation

[CollectionDefinition(Name, DisableParallelization = true)]
public class IntegrationTestCollection : ICollectionFixture<CrdInstaller>
{
    public const string Name = "Integration Tests";
}

public sealed class CrdInstaller : IAsyncLifetime
{
    private List<V1CustomResourceDefinition> _crds = [];

    public async Task InitializeAsync()
    {
        // Transpile CRDs from your entity types
        await using var p = new MlcProvider();
        await p.InitializeAsync();
        _crds = p.Mlc.Transpile(new[] { typeof(V1OperatorIntegrationTestEntity) }).ToList();

        // Install CRDs in the test cluster
        using var client = new Kubernetes(KubernetesClientConfiguration.BuildDefaultConfig());
        foreach (var crd in _crds)
        {
            switch (await client.ApiextensionsV1.ListCustomResourceDefinitionAsync(
                        fieldSelector: $"metadata.name={crd.Name()}"))
            {
                case { Items: [var existing] }:
                    // Update existing CRD
                    crd.Metadata.ResourceVersion = existing.ResourceVersion();
                    await client.ApiextensionsV1.ReplaceCustomResourceDefinitionAsync(crd, crd.Name());
                    break;
                default:
                    // Create new CRD
                    await client.ApiextensionsV1.CreateCustomResourceDefinitionAsync(crd);
                    break;
            }
        }
    }

    public async Task DisposeAsync()
    {
        // Clean up CRDs after tests
        using var client = new Kubernetes(KubernetesClientConfiguration.BuildDefaultConfig());
        foreach (var crd in _crds)
        {
            await client.ApiextensionsV1.DeleteCustomResourceDefinitionAsync(crd.Name());
        }
    }
}

The CrdInstaller is a crucial component that:

1. Transpiles your entity types into CRDs
2. Installs these CRDs in the test cluster
3. Updates existing CRDs if they already exist
4. Cleans up CRDs after tests complete

This ensures that your custom resources are available in the test cluster before running the tests.

Test Base Class

[Collection(IntegrationTestCollection.Name)]
public abstract class IntegrationTestBase : IAsyncLifetime
{
    private IHost? _host;
    protected IServiceProvider Services => _host?.Services ?? throw new InvalidOperationException();

    public virtual async Task InitializeAsync()
    {
        var builder = Host.CreateApplicationBuilder();
        ConfigureHost(builder);
        _host = builder.Build();
        await _host.StartAsync();
    }

    public virtual async Task DisposeAsync()
    {
        if (_host is null) return;
        await _host.StopAsync();
    }

    protected abstract void ConfigureHost(HostApplicationBuilder builder);
}

Invocation Counter

public class InvocationCounter<TEntity> where TEntity : IKubernetesObject<V1ObjectMeta>
{
    private TaskCompletionSource _task = new();
    private readonly ConcurrentQueue<(string Method, TEntity Entity)> _invocations = new();
    public IReadOnlyList<(string Method, TEntity Entity)> Invocations => _invocations.ToList();
    public Task WaitForInvocations => _task.Task;
    public int TargetInvocationCount { get; set; } = 1;

    public void Invocation(TEntity entity, [CallerMemberName] string name = "Invocation")
    {
        _invocations.Enqueue((name, entity));
        if (Invocations.Count >= TargetInvocationCount)
        {
            _task.TrySetResult();
        }
    }
}

Example Tests

The following examples show how KubeOps tests its own functionality. You can use these patterns to test your operator:

Testing Controller Behavior

public class EntityControllerIntegrationTest : IntegrationTestBase
{
    private readonly InvocationCounter<V1OperatorIntegrationTestEntity> _mock = new();
    private readonly IKubernetesClient _client = new KubernetesClient.KubernetesClient();
    private readonly TestNamespaceProvider _ns = new();

    [Fact]
    public async Task Should_Call_Reconcile_On_New_Entity()
    {
        // Create a new entity
        await _client.CreateAsync(new V1OperatorIntegrationTestEntity(
            "test-entity",
            "username",
            _ns.Namespace));

        // Wait for the controller to process it
        await _mock.WaitForInvocations;

        // Verify the controller was called
        _mock.Invocations.Count.Should().Be(1);
        var (method, entity) = _mock.Invocations[0];
        method.Should().Be("ReconcileAsync");
        entity.Name().Should().Be("test-entity");
        entity.Spec.Username.Should().Be("username");
    }

    [Fact]
    public async Task Should_Call_Reconcile_On_Modification_Of_Entity()
    {
        _mock.TargetInvocationCount = 2;

        // Create and then modify an entity
        var result = await _client.CreateAsync(new V1OperatorIntegrationTestEntity(
            "test-entity",
            "username",
            _ns.Namespace));
        result.Spec.Username = "changed";
        await _client.UpdateAsync(result);

        await _mock.WaitForInvocations;

        // Verify both reconciliations
        _mock.Invocations.Count.Should().Be(2);
        _mock.Invocations[0].Entity.Spec.Username.Should().Be("username");
        _mock.Invocations[1].Entity.Spec.Username.Should().Be("changed");
    }

    protected override void ConfigureHost(HostApplicationBuilder builder)
    {
        builder.Services
            .AddSingleton(_mock)
            .AddKubernetesOperator(s => s.Namespace = _ns.Namespace)
            .AddController<TestController, V1OperatorIntegrationTestEntity>();
    }

    private class TestController : IEntityController<V1OperatorIntegrationTestEntity>
    {
        private readonly InvocationCounter<V1OperatorIntegrationTestEntity> _svc;

        public TestController(InvocationCounter<V1OperatorIntegrationTestEntity> svc)
        {
            _svc = svc;
        }

        public Task ReconcileAsync(V1OperatorIntegrationTestEntity entity, CancellationToken cancellationToken)
        {
            _svc.Invocation(entity);
            return Task.CompletedTask;
        }
    }
}

Testing Namespace Isolation

public class NamespacedOperatorIntegrationTest : IntegrationTestBase
{
    private readonly InvocationCounter<V1OperatorIntegrationTestEntity> _mock = new();
    private readonly IKubernetesClient _client = new KubernetesClient.KubernetesClient();
    private readonly TestNamespaceProvider _ns = new();
    private V1Namespace _otherNamespace = null!;

    [Fact]
    public async Task Should_Not_Call_Reconcile_On_Entity_In_Other_Namespace()
    {
        // Create an entity in a different namespace
        await _client.CreateAsync(new V1OperatorIntegrationTestEntity(
            "test-entity2",
            "username",
            _otherNamespace.Name()));

        // Verify the controller wasn't called
        _mock.Invocations.Count.Should().Be(0);
    }

    protected override void ConfigureHost(HostApplicationBuilder builder)
    {
        builder.Services
            .AddSingleton(_mock)
            .AddKubernetesOperator(s => s.Namespace = _ns.Namespace)
            .AddController<TestController, V1OperatorIntegrationTestEntity>();
    }
}

Best Practices

1. Test Isolation:

   • Use unique namespaces for each test
   • Clean up resources after tests
   • Avoid test interdependence

2. Resource Management:

   • Create test-specific resources
   • Use a namespace provider for namespace management
   • Implement proper cleanup in DisposeAsync

3. Test Organization:

   • Group related tests in test classes
   • Use descriptive test names
   • Follow the Arrange-Act-Assert pattern

4. Error Handling:

   • Test both success and failure cases
   • Verify error conditions
   • Check resource cleanup on failure

Common Pitfalls

1. Resource Cleanup:

   • Always clean up created resources
   • Handle cleanup failures gracefully
   • Use IAsyncLifetime for proper setup/teardown

2. Timing Issues:

   • Implement a mechanism to wait for operations
   • Set appropriate timeouts
   • Handle race conditions

3. Namespace Management:

   • Be aware of namespace-scoped operators
   • Test cross-namespace behavior
   • Verify namespace isolation