Sluice / Pub/Sub Setup

Google Cloud Pub/Sub Setup

Configure Sluice to publish messages to Google Cloud Pub/Sub.

Prerequisites

• A GCP project with the Pub/Sub API enabled
• Topics created in Pub/Sub before configuring Sluice (it publishes to existing topics, it does not create them)
• Authentication credentials (see below)

For local development, use the Pub/Sub emulator:

gcloud beta emulators pubsub start --project=my-local-project

The emulator runs on localhost:8085 by default.

Minimal Configuration

listen_addr: "0.0.0.0:8080"

bus:
  type: pubsub
  project_id: "my-gcp-project"

topics:
  my-events:
    target: "my-events-topic"

Publish a message:

curl -X POST http://localhost:8080/v1/publish/my-events \
  -H "Content-Type: application/octet-stream" \
  -H "x-partition-key: order-123" \
  --data-binary '{"event": "created"}'

Response:

{
  "status": "ok",
  "metadata": {
    "message_id": "1234567890"
  }
}

Authentication

Application Default Credentials (recommended)

If no credentials_file is set, Sluice uses ADC. This works automatically on GKE, Cloud Run, GCE (attached service account), and locally via gcloud auth application-default login.

bus:
  type: pubsub
  project_id: "my-gcp-project"
  # No credentials_file, uses ADC

Service account JSON key file

For environments where ADC isn't available:

bus:
  type: pubsub
  project_id: "my-gcp-project"
  credentials_file: "/etc/sluice/sa-key.json"

The service account needs roles/pubsub.publisher on the topics Sluice publishes to.

Emulator

Set the endpoint field to point at the emulator:

bus:
  type: pubsub
  project_id: "my-local-project"
  endpoint: "localhost:8085"

• No authentication required
• Topics must be created via the emulator API before publishing
• project_id can be any string, just match what you used to create topics

Create a topic in the emulator:

curl -X PUT "http://localhost:8085/v1/projects/my-local-project/topics/my-events-topic"

Configuration Reference

Ordering

Pub/Sub supports ordering natively via ordering keys. When you publish with an ordering key header, Sluice passes it through:

curl -X POST http://localhost:8080/v1/publish/my-events \
  -H "x-partition-key: user-123" \
  --data-binary '...'

For end-to-end ordering, the subscription also needs ordering enabled:

gcloud pubsub subscriptions create my-sub \
  --topic=my-events-topic \
  --enable-message-ordering

Sluice provides two ordering layers:

• Gateway-level (actor pool): ensures messages enter Pub/Sub in wire-arrival order per key. Enabled via ordering.enabled: true.
• Pub/Sub-native: delivers messages to subscribers in publication order per ordering key.

For most use cases, enable both. Gateway ordering prevents concurrent HTTP requests from reordering messages before they reach Pub/Sub.

Schema Validation

Bus-native schema fetch requires a license key. Automatic schema loading from GCP topic-attached schemas and schema_poll_interval_secs require a license key. Without one, use schema_override to load schemas from local files.

With Pub/Sub, schemas come from the topic itself. In GCP, you can attach a schema to a Pub/Sub topic. When a topic has an attached schema, Sluice fetches it automatically via the Pub/Sub Admin API. No extra configuration is needed in Sluice -- just attach the schema to your topic in GCP.

For validation to work, the schema must be attached to the topic in GCP. You can attach a schema when creating the topic or add one later:

# Create a schema in GCP
gcloud pubsub schemas create user-events-schema \
  --type=protocol-buffer \
  --definition-file=user_event.proto

# Create a topic with the schema attached
gcloud pubsub topics create user-events \
  --schema=user-events-schema \
  --message-encoding=binary

If no schema is attached to the topic in GCP, Sluice skips schema-level validation for that topic (field-level required_fields checks still run if configured).

Local schema override (dev/testing, free tier)

For local development (including the emulator, which does not support schemas), you can override with local schema files. This works without a license key.

schema_override:
  schemas_dir: "schemas/"

Looks for schemas/{topic-name}.desc (proto), .avsc (avro), or .json. Format is inferred from which file exists. When schema_override is set, it takes priority over any GCP-attached schema.

Publisher Tuning

Requires a license key. Per-topic tuning fields (message_count_threshold, byte_threshold, delay_threshold_ms) are ignored without a license key. All topics use Pub/Sub default batching in the free tier.

Each Pub/Sub topic gets its own publisher client. By default, the publisher uses Pub/Sub's standard batching settings. You can override these per topic by putting tuning fields directly on the topic -- no nesting required:

topics:
  high-throughput:
    message_count_threshold: 500
    byte_threshold: 5242880
    delay_threshold_ms: 50

  low-latency:
    message_count_threshold: 1
    delay_threshold_ms: 1

  default-settings: {}    # uses Pub/Sub defaults

All fields are optional. Only set what you want to change from the Pub/Sub defaults.

A batch is sent as soon as any threshold is reached. For example, if message_count_threshold is 500 and delay_threshold_ms is 50, a batch goes out after 500 messages or 50ms, whichever comes first.

When to tune:

• High throughput -- increase message_count_threshold and byte_threshold to send larger batches. Reduces per-message overhead.
• Low latency -- decrease delay_threshold_ms (or set message_count_threshold: 1) to send messages as quickly as possible.
• Large payloads -- increase byte_threshold if individual messages are large and you want to batch more of them.

Pub/Sub tuning fields are ignored when bus type is kafka.

Config Schema Reference

bus:
  type: pubsub                               # required
  project_id: "my-gcp-project"               # required, GCP project ID
  credentials_file: "/path/to/key.json"      # optional, falls back to ADC
  endpoint: "localhost:8085"                  # optional, for emulator

my-topic:
  target: "bus-native-name"          # optional, defaults to key name
  validation: { ... }                # optional
  ordering: { ... }                  # optional
  message_count_threshold: 500       # optional, default: 100
  byte_threshold: 5242880            # optional, default: 1048576 (1 MB)
  delay_threshold_ms: 50             # optional, default: 10

schema_override:
  schemas_dir: "schemas/"          # looks for {topic}.desc, .avsc, or .json

Differences from Kafka