Skip to content

Add Kafka Connect artifact publish to release process #15212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rmoff wants to merge 1 commit into
base: main
Choose a base branch
from
+93 −5
Open

Add Kafka Connect artifact publish to release process #15212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions kafka-connect/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,10 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
into('doc/') {
from "$projectDir/main/LICENSE"
from "$projectDir/main/NOTICE"
from "$projectDir/main/README.md"
}
into('etc/') {
from "$projectDir/main/etc"
}
into('assets/') {
from "${processResources.destinationDir}/iceberg.png"
Expand All @@ -230,10 +234,6 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
}
}

// there are no Maven artifacts so disable publishing tasks...
project.afterEvaluate {
project.tasks.matching { it.group == 'publishing' }.each {it.enabled = false}
}

tasks.jar.enabled = false

Expand All @@ -249,6 +249,17 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
check.dependsOn integrationTest

assemble.dependsOn distZip, hiveDistZip

// Configure publishing for the distribution zip
afterEvaluate {
publishing {
publications {
apache(MavenPublication) {
artifact distZip
}
}
}
}
}

project(':iceberg-kafka-connect:iceberg-kafka-connect-transforms') {
Expand Down
26 changes: 26 additions & 0 deletions kafka-connect/kafka-connect-runtime/main/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-->

# Apache Iceberg Sink Connector

A Kafka Connect sink connector for writing data from Apache Kafka into Apache Iceberg tables.

Documentation: https://iceberg.apache.org/docs/latest/kafka-connect/

License: Apache License 2.0
27 changes: 27 additions & 0 deletions kafka-connect/kafka-connect-runtime/main/etc/iceberg-sink.properties.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Iceberg Sink Connector - Example Configuration
# https://iceberg.apache.org/docs/latest/kafka-connect/
name=iceberg-sink
connector.class=org.apache.iceberg.connect.IcebergSinkConnector
tasks.max=1
topics=events
iceberg.tables=db.table_name
iceberg.catalog.type=rest
iceberg.catalog.uri=https://your-catalog-uri
iceberg.catalog.warehouse=your-warehouse
2 changes: 1 addition & 1 deletion kafka-connect/kafka-connect-runtime/src/main/resources/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"logo": "assets/iceberg.png",
"type": "organization",
"url": "https://iceberg.apache.org",
"username": "iceberg"
"username": "apache"
},

"support": {
Expand Down
24 changes: 24 additions & 0 deletions site/docs/how-to-release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,11 @@ To build and publish the convenience binaries, run the `dev/stage-binaries.sh` s
dev/stage-binaries.sh
```

!!! Note
The staged binaries include the Kafka Connect runtime distribution
(`iceberg-kafka-connect-runtime-<VERSION>.zip`), which bundles the connector
with all required dependencies for Kafka Connect plugin installation.

Next, you need to close the staging repository:

1. Go to [Nexus](https://repository.apache.org/) and log in
Expand Down Expand Up @@ -297,6 +302,25 @@ Java artifacts are available from Maven Central.
Thanks to everyone for contributing!
```

#### Confluent Marketplace Submission

After the release is published to Maven Central, the Kafka Connect runtime distribution
can be [submitted to](https://docs.confluent.io/platform/current/connect/confluent-hub/contributing.html)
[Confluent Marketplace](https://www.confluent.io/hub/) for broader distribution.

To submit to Confluent Hub:

1. [Contact the Confluent Hub team](mailto:confluent-hub@confluent.io) with the Maven Central URL for the distribution zip:

```
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-kafka-connect-runtime/<VERSION>/iceberg-kafka-connect-runtime-<VERSION>.zip
```

2. Confluent will publish to the Marketplace

!!! Note
This step is optional and can be performed by any PMC member after the release is finalized.

#### Update revapi

Create a PR in the `iceberg` repo to make revapi run on the new release. For an example see [this PR](https://github.com/apache/iceberg/pull/6275).
Expand Down