[aerogear-dev] GSoC week 11 update

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[aerogear-dev] GSoC week 11 update

Dimitra
Hi guys,

It's week 11 of GSoC, which means we only have about 3 weeks left until the end! Here's a quick summary of what we've been doing: 

* Right now I'm currently working on implementing custom object serialisers and deserialisers to give us the ability to produce whatever we want to our topics. 
It's been a bit of a bumpy ride so far but Matthias just did a PR which resolves some of the issues we were having with generics. We'll hopefully have this sorted by the end of the week, so we can start using it for the producers/consumers/streams that we implement.

* Polina has already done AGPUSH-2159 related to our first "custom" producer but hasn't sent a PR off yet, given the aforementioned issues. She's also currently working on getting a fancier test environment up for all of our future consumers and producers :) 

* Since we have a very short amount of time left after that, we'll jump right into some more implementation with AGPUSH-2165 and AGPUSH-2166, which we'll be working on in our next sprint. 

That's it, just wanted to update on what's going on. 
Thanks for the feedback and help, we always appreciate it! 

Dimitra

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [aerogear-dev] GSoC week 11 update

Matthias Wessendorf
Hey Dimitra,

thanks a lot for the updates! It's really cool to see the progress on the project. Also struggles and issues are part of the job, so it's good to see you run into those, and are able to sort things ;-)

Keep it up!

On Tue, Aug 8, 2017 at 10:42 PM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys,

It's week 11 of GSoC, which means we only have about 3 weeks left until the end! Here's a quick summary of what we've been doing: 

* Right now I'm currently working on implementing custom object serialisers and deserialisers to give us the ability to produce whatever we want to our topics. 
It's been a bit of a bumpy ride so far but Matthias just did a PR which resolves some of the issues we were having with generics. We'll hopefully have this sorted by the end of the week, so we can start using it for the producers/consumers/streams that we implement.

* Polina has already done AGPUSH-2159 related to our first "custom" producer but hasn't sent a PR off yet, given the aforementioned issues. She's also currently working on getting a fancier test environment up for all of our future consumers and producers :) 

* Since we have a very short amount of time left after that, we'll jump right into some more implementation with AGPUSH-2165 and AGPUSH-2166, which we'll be working on in our next sprint. 

That's it, just wanted to update on what's going on. 
Thanks for the feedback and help, we always appreciate it! 

Dimitra

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev



--

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [aerogear-dev] GSoC week 11 update

Dimitra
In reply to this post by Dimitra
Hi guys, here's the PR for the generic ser/des: https://github.com/matzew/kafka-cdi/pull/17

As far as testing goes, there are a few issues with abstract classes obviously, and since most of the objects we'll be producing contain 'Variant' (which is an abstract class) we'll have to come up with a nice solution for that - we're open to ideas! 

Right now it won't work "out of the box" with Polina's PR due to the List<Variant> in the PushApplication class: 

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.jboss.aerogear.unifiedpush.api.Variant, problem: abstract types either need to be mapped to concrete types, have custom deserializer, or be instantiated with additional type information

 at [Source: [B@d4163ef; line: 1, column: 229] (through reference chain: org.jboss.aerogear.unifiedpush.api.PushApplication["variants"])



Two other things: 


* Bad things will happen if objects of incorrect type are pushed to the same topic. This will definitely have to be addressed by enforcing some basic security (AGPUSH-2109)


* We'll have to make sure our logs are a little more "informative" as we can't log whole objects obviously for readability. Right now, even with just two consumers, the log is quite overwhelming (i.e. consumer configs etc) 


On Tue, Aug 8, 2017 at 9:42 PM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys,

It's week 11 of GSoC, which means we only have about 3 weeks left until the end! Here's a quick summary of what we've been doing: 

* Right now I'm currently working on implementing custom object serialisers and deserialisers to give us the ability to produce whatever we want to our topics. 
It's been a bit of a bumpy ride so far but Matthias just did a PR which resolves some of the issues we were having with generics. We'll hopefully have this sorted by the end of the week, so we can start using it for the producers/consumers/streams that we implement.

* Polina has already done AGPUSH-2159 related to our first "custom" producer but hasn't sent a PR off yet, given the aforementioned issues. She's also currently working on getting a fancier test environment up for all of our future consumers and producers :) 

* Since we have a very short amount of time left after that, we'll jump right into some more implementation with AGPUSH-2165 and AGPUSH-2166, which we'll be working on in our next sprint. 

That's it, just wanted to update on what's going on. 
Thanks for the feedback and help, we always appreciate it! 

Dimitra


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [aerogear-dev] GSoC week 11 update

Matthias Wessendorf


On Wed, Aug 9, 2017 at 10:35 AM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys, here's the PR for the generic ser/des: https://github.com/matzew/kafka-cdi/pull/17

As far as testing goes, there are a few issues with abstract classes obviously, and since most of the objects we'll be producing contain 'Variant' (which is an abstract class) we'll have to come up with a nice solution for that - we're open to ideas! 

not sure I follow :-) but why not using one of the concrete impls of it ? 
 

Right now it won't work "out of the box" with Polina's PR due to the List<Variant> in the PushApplication class: 

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.jboss.aerogear.unifiedpush.api.Variant, problem: abstract types either need to be mapped to concrete types, have custom deserializer, or be instantiated with additional type information

 at [Source: [B@d4163ef; line: 1, column: 229] (through reference chain: org.jboss.aerogear.unifiedpush.api.PushApplication["variants"])




do we need to define some mapping in UPS?
 

Two other things: 


* Bad things will happen if objects of incorrect type are pushed to the same topic.


when or how ? any code/pointers?
 

This will definitely have to be addressed by enforcing some basic security (AGPUSH-2109)



* We'll have to make sure our logs are a little more "informative" as we can't log whole objects obviously for readability.

+1
 

Right now, even with just two consumers, the log is quite overwhelming (i.e. consumer configs etc) 


we could make the kafka own logging quiet ;-) That's IMO the verbose one, which is sometimes handy.

But in WildFly we could limit those to WARN/ERROR
 

On Tue, Aug 8, 2017 at 9:42 PM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys,

It's week 11 of GSoC, which means we only have about 3 weeks left until the end! Here's a quick summary of what we've been doing: 

* Right now I'm currently working on implementing custom object serialisers and deserialisers to give us the ability to produce whatever we want to our topics. 
It's been a bit of a bumpy ride so far but Matthias just did a PR which resolves some of the issues we were having with generics. We'll hopefully have this sorted by the end of the week, so we can start using it for the producers/consumers/streams that we implement.

* Polina has already done AGPUSH-2159 related to our first "custom" producer but hasn't sent a PR off yet, given the aforementioned issues. She's also currently working on getting a fancier test environment up for all of our future consumers and producers :) 

* Since we have a very short amount of time left after that, we'll jump right into some more implementation with AGPUSH-2165 and AGPUSH-2166, which we'll be working on in our next sprint. 

That's it, just wanted to update on what's going on. 
Thanks for the feedback and help, we always appreciate it! 

Dimitra


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev



--

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [aerogear-dev] GSoC week 11 update

Dimitra
Thanks Matthias! :) 


On Wed, Aug 9, 2017 at 9:46 AM, Matthias Wessendorf <[hidden email]> wrote:


On Wed, Aug 9, 2017 at 10:35 AM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys, here's the PR for the generic ser/des: https://github.com/matzew/kafka-cdi/pull/17

As far as testing goes, there are a few issues with abstract classes obviously, and since most of the objects we'll be producing contain 'Variant' (which is an abstract class) we'll have to come up with a nice solution for that - we're open to ideas! 

not sure I follow :-) but why not using one of the concrete impls of it ? 

Because then we'd have to define a producer/consumer for every concrete implementation, like SimpleKafkaProducer<String, AndroidVariant>, SimpleKafkaProducer<String, AdmVariant> etc.. I think that would be extremely inefficient 
 
 

Right now it won't work "out of the box" with Polina's PR due to the List<Variant> in the PushApplication class: 

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.jboss.aerogear.unifiedpush.api.Variant, problem: abstract types either need to be mapped to concrete types, have custom deserializer, or be instantiated with additional type information

 at [Source: [B@d4163ef; line: 1, column: 229] (through reference chain: org.jboss.aerogear.unifiedpush.api.PushApplication["variants"])




do we need to define some mapping in UPS?

What do you mean? 
 
 

Two other things: 


* Bad things will happen if objects of incorrect type are pushed to the same topic.


when or how ? any code/pointers?
 

This will definitely have to be addressed by enforcing some basic security (AGPUSH-2109)



* We'll have to make sure our logs are a little more "informative" as we can't log whole objects obviously for readability.

+1
 

Right now, even with just two consumers, the log is quite overwhelming (i.e. consumer configs etc) 


we could make the kafka own logging quiet ;-) That's IMO the verbose one, which is sometimes handy.

But in WildFly we could limit those to WARN/ERROR

+1 
It's handy for development but otherwise we should limit it I think
 
 

On Tue, Aug 8, 2017 at 9:42 PM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys,

It's week 11 of GSoC, which means we only have about 3 weeks left until the end! Here's a quick summary of what we've been doing: 

* Right now I'm currently working on implementing custom object serialisers and deserialisers to give us the ability to produce whatever we want to our topics. 
It's been a bit of a bumpy ride so far but Matthias just did a PR which resolves some of the issues we were having with generics. We'll hopefully have this sorted by the end of the week, so we can start using it for the producers/consumers/streams that we implement.

* Polina has already done AGPUSH-2159 related to our first "custom" producer but hasn't sent a PR off yet, given the aforementioned issues. She's also currently working on getting a fancier test environment up for all of our future consumers and producers :) 

* Since we have a very short amount of time left after that, we'll jump right into some more implementation with AGPUSH-2165 and AGPUSH-2166, which we'll be working on in our next sprint. 

That's it, just wanted to update on what's going on. 
Thanks for the feedback and help, we always appreciate it! 

Dimitra


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev



--

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [aerogear-dev] GSoC week 11 update

Matthias Wessendorf


On Wed, Aug 9, 2017 at 11:30 AM, Dimitra Zuccarelli <[hidden email]> wrote:
Thanks Matthias! :) 


On Wed, Aug 9, 2017 at 9:46 AM, Matthias Wessendorf <[hidden email]> wrote:


On Wed, Aug 9, 2017 at 10:35 AM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys, here's the PR for the generic ser/des: https://github.com/matzew/kafka-cdi/pull/17

As far as testing goes, there are a few issues with abstract classes obviously, and since most of the objects we'll be producing contain 'Variant' (which is an abstract class) we'll have to come up with a nice solution for that - we're open to ideas! 

not sure I follow :-) but why not using one of the concrete impls of it ? 

Because then we'd have to define a producer/consumer for every concrete implementation,

how about "<? extends Variant>" ?
 
like SimpleKafkaProducer<String, AndroidVariant>, SimpleKafkaProducer<String, AdmVariant> etc.. I think that would be extremely inefficient 


yeah, perhaps we just route all to one topic, and use some streaming fu to dispatch them to their actual origins ?
 
 
 

Right now it won't work "out of the box" with Polina's PR due to the List<Variant> in the PushApplication class: 

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.jboss.aerogear.unifiedpush.api.Variant, problem: abstract types either need to be mapped to concrete types, have custom deserializer, or be instantiated with additional type information

 at [Source: [B@d4163ef; line: 1, column: 229] (through reference chain: org.jboss.aerogear.unifiedpush.api.PushApplication["variants"])




do we need to define some mapping in UPS?

What do you mean? 
 
 

Two other things: 


* Bad things will happen if objects of incorrect type are pushed to the same topic.


when or how ? any code/pointers?
 

This will definitely have to be addressed by enforcing some basic security (AGPUSH-2109)



* We'll have to make sure our logs are a little more "informative" as we can't log whole objects obviously for readability.

+1
 

Right now, even with just two consumers, the log is quite overwhelming (i.e. consumer configs etc) 


we could make the kafka own logging quiet ;-) That's IMO the verbose one, which is sometimes handy.

But in WildFly we could limit those to WARN/ERROR

+1 
It's handy for development but otherwise we should limit it I think
 
 

On Tue, Aug 8, 2017 at 9:42 PM, Dimitra Zuccarelli <[hidden email]> wrote:
Hi guys,

It's week 11 of GSoC, which means we only have about 3 weeks left until the end! Here's a quick summary of what we've been doing: 

* Right now I'm currently working on implementing custom object serialisers and deserialisers to give us the ability to produce whatever we want to our topics. 
It's been a bit of a bumpy ride so far but Matthias just did a PR which resolves some of the issues we were having with generics. We'll hopefully have this sorted by the end of the week, so we can start using it for the producers/consumers/streams that we implement.

* Polina has already done AGPUSH-2159 related to our first "custom" producer but hasn't sent a PR off yet, given the aforementioned issues. She's also currently working on getting a fancier test environment up for all of our future consumers and producers :) 

* Since we have a very short amount of time left after that, we'll jump right into some more implementation with AGPUSH-2165 and AGPUSH-2166, which we'll be working on in our next sprint. 

That's it, just wanted to update on what's going on. 
Thanks for the feedback and help, we always appreciate it! 

Dimitra


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev



--

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev



--

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Loading...