How it works
One gateway. Every topic governed.
Configure
Define topics in YAML with schema sources, validation rules, ordering keys, and producer settings. One config file controls your entire publish surface. Topics map logical names to bus-native targets.
Validate
Payloads are checked against proto and Avro schemas using partial deserialization. Only the fields that matter are decoded. Bad data gets a 400 before it ever reaches the bus. Required fields, type checks, enum constraints.
Publish
Messages are dispatched to the bus with write-time ordering guarantees per key. Per-topic producers isolate throughput-sensitive topics. Kafka, Pub/Sub, or any backend behind the same HTTP API.
Bus-agnostic by design.
Your applications publish to a single HTTP endpoint. Sluice handles the bus-specific details: partitioning for Kafka, ordering keys for Pub/Sub. Switch backends or add new ones. Client code stays the same.
Get started
Apache Kafka. Per-topic producers, idempotent writes, partition key routing
Google Pub/Sub. Native ordering keys, ADC auth, emulator support
More coming. Kinesis, EventHub. Same API, same validation, same ordering
Capabilities
Data quality enforcement at the publish boundary.
Schema validation
- Proto wire-format tag scanning, no full deserialization
- Avro schema-guided projection reading
- Required field checks, type validation, enum constraints
- Local schema files included free; registry sync requires Pro
Write-time ordering
- Per-key actor pool guarantees wire-arrival order
- Lazy actor spawn, scales with active key cardinality
- Idle eviction keeps memory bounded
- Works across Kafka, Pub/Sub, and future backends
Per-topic control
- Topic-level validation, ordering, and routing config
- Logical topic names decoupled from bus-native targets
- Dedicated Kafka producers and Pub/Sub batching overrides (Pro)
- One default producer handles the long tail
Operational
- Single static binary. Deploy as container, sidecar, or standalone
- YAML config with environment variable overrides
- Health checks, structured JSON logging, graceful shutdown
- Prometheus /metrics + request ID correlation included free
- OTLP distributed tracing with an active Pro entitlement
No bad data hits your topics.
Partial deserialization means Sluice validates payloads at gateway scale without the overhead of full deserialization. Proto's tag-based format is scanned in a single pass. Avro's positional format is navigated with a schema-guided projection reader. Only the fields that matter are decoded.
Proto validation
Wire-format tag scanning reads field numbers and types directly from the binary payload. Required fields are checked, values are type-verified, and everything else is skipped at wire speed.
Avro validation
Schema-guided projection uses the writer schema to navigate to required fields and verify they are present and non-null, without materializing the entire record.
JSON schema validation
JSON payloads are validated against a JSON Schema definition. Required fields are checked for presence and non-null values. Schema is loaded from the registry or local files.
Architecture
Runs in your infrastructure.
- HTTP clients
- Microservices
- Mobile / IoT
- Webhooks
- Apache Kafka
- Google Pub/Sub
- Kinesis (coming soon)
- EventHub (coming soon)
Data never leaves your network. The gateway runs in your infrastructure as a container or sidecar.
Install
Get the gateway.
helm install sluice oci://ghcr.io/data-tier/charts/sluice --version 0.2.3 -f values.yamldocker run -v ./sluice.yaml:/etc/sluice/sluice.yaml \
-p 8080:8080 ghcr.io/data-tier/sluice:0.5.0services:
sluice:
image: ghcr.io/data-tier/sluice:0.5.0
ports: ["8080:8080"]
volumes: ["./sluice.yaml:/etc/sluice/sluice.yaml"]Quick start.
Configure your topics, start the gateway, and publish via HTTP.
Configure sluice.yaml
bus:
type: kafka
bootstrap_servers: "localhost:9092"
topics:
my-events: {}bus:
type: pubsub
project_id: "my-gcp-project"
topics:
my-events: {}Start
docker run -v ./sluice.yaml:/etc/sluice/sluice.yaml -p 8080:8080 ghcr.io/data-tier/sluice:0.5.0
Publish
curl -X POST http://localhost:8080/v1/publish/my-events -d '{"hello":"world"}'
Pricing
Free core. One-time 30-day Pro trial.
Run the gateway indefinitely with local schemas, validation, ordering, and basic observability. Creating an organisation starts one Pro trial that expires after 30 days and automatically returns every gateway to Free unless you subscribe.
- Publish API (Kafka + Pub/Sub)
- Proto, Avro, and JSON validation
- Per-key write-time ordering
- Local schema files
- Health check endpoint
- Prometheus /metrics endpoint
- Request and trace ID correlation
- Structured JSON logs
- Graceful shutdown with drain timeout
- Everything in Free
- Confluent Schema Registry sync (Kafka)
- GCP topic-attached schema sync (Pub/Sub)
- Schema hot-reload (polling + file watch)
- Per-topic producer tuning
- Kafka dedicated producers
- Pub/Sub batching overrides
- OTLP distributed tracing
- One managed production cluster
- Acknowledged configuration rollouts
- Fleet health and 7-day telemetry
- Everything in Pro
- Topic management, fleet telemetry, and acknowledged config rollouts
- RBAC, teams, OIDC SSO, scoped API keys, and retained audit logging
- Health-aware active/failover routing across named gateway clusters
- HA Kubernetes deployment with external PostgreSQL integration
- Dedicated support and contractual SLA options
Try Sluice Pro
Put Pro to work for 30 days.
Evaluate registry-backed schemas, live configuration, production tuning, distributed tracing, and fleet visibility in your own environment. When the trial ends, your gateways continue on the Free plan without interrupting publishing.
Enterprise control
Operate a gateway fleet without putting the control plane in the data path.
Assign topics to production clusters, observe gateway and topic health, track each configuration revision through apply or failure, and fail over by health and priority. Customer payloads continue directly from gateways to Kafka or Pub/Sub.
Bad data shouldn't reach your topics.
Deploy the gateway in your infra and start publishing with validation and ordering.