[aerogear-dev] Prepping for Data Sync in AeroGear.js

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[aerogear-dev] Prepping for Data Sync in AeroGear.js

Kris Borchers
Hey all,

Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there.

Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning.

  • New record is added to store, status is set to NEW and a UUID is generated
  • Record is updated, status is set to MODIFIED
  • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind)
  • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced
    • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy

There's probably more to this but these are my original thoughts right now.

Kris

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] Prepping for Data Sync in AeroGear.js

Sebastien Blanc
Hi,
Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me.
But great challenges to come like dealing multiple edits of the same record by different clients :)
About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process. 
On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come.

Seb
Envoyé de mon iPhone

Le Oct 10, 2012 à 17:31, Kris Borchers <[hidden email]> a écrit :

Hey all,

Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there.

Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning.

  • New record is added to store, status is set to NEW and a UUID is generated
  • Record is updated, status is set to MODIFIED
  • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind)
  • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced
    • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy

There's probably more to this but these are my original thoughts right now.

Kris
_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
qmx
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] Prepping for Data Sync in AeroGear.js

qmx
Administrator

On Oct 10, 2012, at 1:07 PM, Sébastien Blanc wrote:

> Hi,
> Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me.
> But great challenges to come like dealing multiple edits of the same record by different clients :)
One strategy that I've seen in the wild is having smaller records, (a contact entity could potentially have a timestamp for each field, to aid during the merge)

I know it's pretty aggressive and wasteful - just food for thought

> About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process.
> On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come.
>
> Seb
> Envoyé de mon iPhone
>
> Le Oct 10, 2012 à 17:31, Kris Borchers <[hidden email]> a écrit :
>
>> Hey all,
>>
>> Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there.
>>
>> Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning.
>>
>> • New record is added to store, status is set to NEW and a UUID is generated
>> • Record is updated, status is set to MODIFIED
>> • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind)
>> • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced
>> • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy
>>
>> There's probably more to this but these are my original thoughts right now.
>>
>> Kris
>> _______________________________________________
>> aerogear-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
> _______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev

-- qmx


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] Prepping for Data Sync in AeroGear.js

Sebastien Blanc
I think we should also take a look at the "slowly changing dimension " theories :
http://en.m.wikipedia.org/wiki/Slowly_changing_dimension#
In particularly type VI

Envoyé de mon iPhone

Le Oct 10, 2012 à 18:38, Douglas Campos <[hidden email]> a écrit :

>
> On Oct 10, 2012, at 1:07 PM, Sébastien Blanc wrote:
>
>> Hi,
>> Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me.
>> But great challenges to come like dealing multiple edits of the same record by different clients :)
> One strategy that I've seen in the wild is having smaller records, (a contact entity could potentially have a timestamp for each field, to aid during the merge)
>
> I know it's pretty aggressive and wasteful - just food for thought
>
>> About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process.
>> On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come.
>>
>> Seb
>> Envoyé de mon iPhone
>>
>> Le Oct 10, 2012 à 17:31, Kris Borchers <[hidden email]> a écrit :
>>
>>> Hey all,
>>>
>>> Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there.
>>>
>>> Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning.
>>>
>>>    • New record is added to store, status is set to NEW and a UUID is generated
>>>    • Record is updated, status is set to MODIFIED
>>>    • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind)
>>>    • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced
>>>        • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy
>>>
>>> There's probably more to this but these are my original thoughts right now.
>>>
>>> Kris
>>> _______________________________________________
>>> aerogear-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>> _______________________________________________
>> aerogear-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
> -- qmx
>
>
> _______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] Prepping for Data Sync in AeroGear.js

Burr Sutter
Ideally our framework would be highly configurable:
- collision detection and resolution at the row/tuple/object-level
- or at the column/property level
- sync-synchronization - block the user thread
- async-synchronization - perform on a "background" thread
- utilization push notifications to cause the client-app to walk up and request the changes

It is relatively rare to have a collision (two users or two batch processes) updating the same record simultaneously but it is a use case that should be addressed.   There was one technology I worked with in the past that took a hash of the before "image" of the record and submit that back with the deltas - relatively small over-the-wire packet and allowed easy detection to see if another data element changed.

On Oct 10, 2012, at 12:53 PM, Sébastien Blanc wrote:

> I think we should also take a look at the "slowly changing dimension " theories :
> http://en.m.wikipedia.org/wiki/Slowly_changing_dimension#
> In particularly type VI
>
> Envoyé de mon iPhone
>
> Le Oct 10, 2012 à 18:38, Douglas Campos <[hidden email]> a écrit :
>
>>
>> On Oct 10, 2012, at 1:07 PM, Sébastien Blanc wrote:
>>
>>> Hi,
>>> Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me.
>>> But great challenges to come like dealing multiple edits of the same record by different clients :)
>> One strategy that I've seen in the wild is having smaller records, (a contact entity could potentially have a timestamp for each field, to aid during the merge)
>>
>> I know it's pretty aggressive and wasteful - just food for thought
>>
>>> About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process.
>>> On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come.
>>>
>>> Seb
>>> Envoyé de mon iPhone
>>>
>>> Le Oct 10, 2012 à 17:31, Kris Borchers <[hidden email]> a écrit :
>>>
>>>> Hey all,
>>>>
>>>> Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there.
>>>>
>>>> Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning.
>>>>
>>>>   • New record is added to store, status is set to NEW and a UUID is generated
>>>>   • Record is updated, status is set to MODIFIED
>>>>   • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind)
>>>>   • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced
>>>>       • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy
>>>>
>>>> There's probably more to this but these are my original thoughts right now.
>>>>
>>>> Kris
>>>> _______________________________________________
>>>> aerogear-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>> _______________________________________________
>>> aerogear-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>> -- qmx
>>
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
> _______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] Prepping for Data Sync in AeroGear.js

Kris Borchers

On Oct 10, 2012, at 3:22 PM, Burr Sutter <[hidden email]> wrote:

> Ideally our framework would be highly configurable:
> - collision detection and resolution at the row/tuple/object-level
> - or at the column/property level
> - sync-synchronization - block the user thread
> - async-synchronization - perform on a "background" thread
> - utilization push notifications to cause the client-app to walk up and request the changes
>
> It is relatively rare to have a collision (two users or two batch processes) updating the same record simultaneously but it is a use case that should be addressed.   There was one technology I worked with in the past that took a hash of the before "image" of the record and submit that back with the deltas - relatively small over-the-wire packet and allowed easy detection to see if another data element changed.

That's actually a good idea. Easy enough to stringify and hash a chunk of data and send that along with the changed record. Then the server does the same hash before an update and can report collisions. I like that a lot actually.

>
> On Oct 10, 2012, at 12:53 PM, Sébastien Blanc wrote:
>
>> I think we should also take a look at the "slowly changing dimension " theories :
>> http://en.m.wikipedia.org/wiki/Slowly_changing_dimension#
>> In particularly type VI
>>
>> Envoyé de mon iPhone
>>
>> Le Oct 10, 2012 à 18:38, Douglas Campos <[hidden email]> a écrit :
>>
>>>
>>> On Oct 10, 2012, at 1:07 PM, Sébastien Blanc wrote:
>>>
>>>> Hi,
>>>> Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me.
>>>> But great challenges to come like dealing multiple edits of the same record by different clients :)
>>> One strategy that I've seen in the wild is having smaller records, (a contact entity could potentially have a timestamp for each field, to aid during the merge)
>>>
>>> I know it's pretty aggressive and wasteful - just food for thought
>>>
>>>> About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process.
>>>> On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come.
>>>>
>>>> Seb
>>>> Envoyé de mon iPhone
>>>>
>>>> Le Oct 10, 2012 à 17:31, Kris Borchers <[hidden email]> a écrit :
>>>>
>>>>> Hey all,
>>>>>
>>>>> Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there.
>>>>>
>>>>> Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning.
>>>>>
>>>>>  • New record is added to store, status is set to NEW and a UUID is generated
>>>>>  • Record is updated, status is set to MODIFIED
>>>>>  • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind)
>>>>>  • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced
>>>>>      • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy
>>>>>
>>>>> There's probably more to this but these are my original thoughts right now.
>>>>>
>>>>> Kris
>>>>> _______________________________________________
>>>>> aerogear-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>>> _______________________________________________
>>>> aerogear-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>>
>>> -- qmx
>>>
>>>
>>> _______________________________________________
>>> aerogear-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
> _______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] Prepping for Data Sync in AeroGear.js

Kris Borchers
OK so the first pass at just tracking data status is here

https://github.com/aerogear/aerogear-js/commit/b110eafa93ce9ea39febbcea72fdc04c28ee50c3

Please take a look and let me know what you think. Not much but a start. :)

On Oct 10, 2012, at 3:25 PM, Kris Borchers <[hidden email]> wrote:

>
> On Oct 10, 2012, at 3:22 PM, Burr Sutter <[hidden email]> wrote:
>
>> Ideally our framework would be highly configurable:
>> - collision detection and resolution at the row/tuple/object-level
>> - or at the column/property level
>> - sync-synchronization - block the user thread
>> - async-synchronization - perform on a "background" thread
>> - utilization push notifications to cause the client-app to walk up and request the changes
>>
>> It is relatively rare to have a collision (two users or two batch processes) updating the same record simultaneously but it is a use case that should be addressed.   There was one technology I worked with in the past that took a hash of the before "image" of the record and submit that back with the deltas - relatively small over-the-wire packet and allowed easy detection to see if another data element changed.
>
> That's actually a good idea. Easy enough to stringify and hash a chunk of data and send that along with the changed record. Then the server does the same hash before an update and can report collisions. I like that a lot actually.
>>
>> On Oct 10, 2012, at 12:53 PM, Sébastien Blanc wrote:
>>
>>> I think we should also take a look at the "slowly changing dimension " theories :
>>> http://en.m.wikipedia.org/wiki/Slowly_changing_dimension#
>>> In particularly type VI
>>>
>>> Envoyé de mon iPhone
>>>
>>> Le Oct 10, 2012 à 18:38, Douglas Campos <[hidden email]> a écrit :
>>>
>>>>
>>>> On Oct 10, 2012, at 1:07 PM, Sébastien Blanc wrote:
>>>>
>>>>> Hi,
>>>>> Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me.
>>>>> But great challenges to come like dealing multiple edits of the same record by different clients :)
>>>> One strategy that I've seen in the wild is having smaller records, (a contact entity could potentially have a timestamp for each field, to aid during the merge)
>>>>
>>>> I know it's pretty aggressive and wasteful - just food for thought
>>>>
>>>>> About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process.
>>>>> On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come.
>>>>>
>>>>> Seb
>>>>> Envoyé de mon iPhone
>>>>>
>>>>> Le Oct 10, 2012 à 17:31, Kris Borchers <[hidden email]> a écrit :
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there.
>>>>>>
>>>>>> Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning.
>>>>>>
>>>>>> • New record is added to store, status is set to NEW and a UUID is generated
>>>>>> • Record is updated, status is set to MODIFIED
>>>>>> • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind)
>>>>>> • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced
>>>>>>     • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy
>>>>>>
>>>>>> There's probably more to this but these are my original thoughts right now.
>>>>>>
>>>>>> Kris
>>>>>> _______________________________________________
>>>>>> aerogear-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>>>> _______________________________________________
>>>>> aerogear-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>>>
>>>> -- qmx
>>>>
>>>>
>>>> _______________________________________________
>>>> aerogear-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>>
>>> _______________________________________________
>>> aerogear-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>>
>>
>> _______________________________________________
>> aerogear-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
>
> _______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev