Hi, anyone know if there is a plan to
Hi, anyone know if there is a plan to support S3 for pipelines (beta), currently only R2 is sopported, I wanted to try it out with our Confluent Kafka, but they don't support R2 only AWS s3.
From what I undersand Pipelines are similar to Firehose, But I would prefer to stay on Cloudfalre.
8 Replies
Eventually we'll offer more sinks - does the Kafka S3 connector not support S3-compliant APIs/endpoints?
https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html#using-non-aws-storage-providers (though not sure if this works with the managed version)
Amazon S3 Sink Connector for Confluent Platform | Confluent Documen...
Get started with the Amazon S3 Sink connector for Confluent Platform.
Hi @Marc thank you for reply, unfortunately no, the plugin is only for AWS S3, they have separate plugins for Google, azure, but no R2
I’m currently in talks with Confluent they have hadrcoded our R2 endpoint, but no luck passing authentication, so we are back and forth of testing it
hmm I find that odd - standby going to try something
can you share the kafka connector config you tried with?
(can be anaonymized of course)
oh are you talking about Tiered Storage or Kafka Connect? I should have asked first
Hi @Marc
The above URL is for S3 Sink and they do support custom S3 provider like R2.
But Sink is from Kafka -> S3/R2
What we are looking for is R2 Source. and there is no way to change the endpoint
Confluent Hub: Apache Kafka Connectors for Streaming Data
Discover 200+ expert-built Apache Kafka connectors for seamless, real-time data streaming and integration. Connect with MongoDB, AWS S3, Snowflake, and more.
Are you self-managing the connector? Out of the box you can specify the
store.url which should work - but if you're using the Confluent Cloud managed version it does not allow for a custom endpoint https://docs.confluent.io/kafka-connectors/s3-source/current/configuration_options.htmlYeah we are using Cloud Managed connector I will check if we can add self managed