0

i'm trying to achieve

{
"source": "NEWS",
"metadata": {
"publishTime": "02/06/2019 09:56:24.317",
"channel": "paper",
"status":"active"
},
"Data": {
"NAME": 67,
"GENDER": "MALE",
    ...
}
}

but i'm stuck with this

{
    "Data": {
        "NAME": 67,
        "GENDER": "MALE",
    ...

    },
    "source": "NEWS",
    "metadata": "{\"channel\":'paper'}"
}

and below are my connector.properties

name=source-sqlserver-user
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:sqlserver://localhost:1433;database=testing;username=xyz;password=xyz;
table.whitelist=Tickets
mode=incrementing 
incrementing.column.name=Ticket_id
validate.non.null=false
topic.prefix=my-mssql-
transforms=MakeMap123,value,extract,InsertSourceDetails,xyz
transforms.MakeMap123.type=org.apache.kafka.connect.transforms.HoistField$Value
transforms.MakeMap123.field=Data
transforms.value.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.value.fields=Ticket_id
transforms.extract.type=org.apache.kafka.connect.transforms.ExtractField$Value
transforms.extract.field=Ticket_id
#transforms.InsertTopic.type=org.apache.kafka.connect.transforms.InsertField$Value
#transforms.InsertTopic.topic.field=messagetopic
eetransforms.InsertSourceDetails.type=org.apache.kafka.connect.transforms.InsertField$Value
transforms.InsertSourceDetails.static.field=source
transforms.InsertSourceDetails.static.value=NEWS
transforms.xyz.type=org.apache.kafka.connect.transforms.InsertField$Value
transforms.xyz.static.field=metadata
transforms.xyz.static.value={"channel":'paper'}

In here source:NEWS is static field and works as excepted but, is there any possiblilty that we extract the few columns of the same table(all the columns belongs to a single table) and put it as another json key(metadata in my case) as shown above. and yes, i tried ValueToKey and ExtractField$Value but ValueToKey throws NPE

Is this possible to achieve with kafka connect Transformation? or am i missing something?? or should i use custom avro schema? any examples for custom avro schema is much appreciated.

  • Your metadata field is a string. So, there isn't much Connect can do to parse and expand that to an object. At least there isn't a built in transform for it. And you've shown no ticket id field, so that's why there's an NPE – cricket_007 Apr 11 at 15:25
  • @cricket_007 My bad, i forgot to add ticket id field in the json (if that what your referring to) and enabling schema got me resolved NPE but still i'm not able to use ValueToKey and here is example i tried i.imgur.com/6mmmJ1e.png – Clover Apr 12 at 6:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.