[aerogear-dev] sync

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[aerogear-dev] sync

Erik Jan de Wit
So we already agreed that CouchDB has some nice features, but why wouldn’t we just use pouchdb for javascript and touchdb? I’ve just tested the javascript and it syncs with the server perfectly, even if there are some things missing we could add them because these projects are open source.

Cheers,
        Erik Jan
_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Sebastien Blanc
Sure, PouchDB is nice, and we had already plans to make PouchDB datamanager adapter (I think Luke even started on this). But for sync, in general, don't we want to be database agnostic ? 


On Fri, Dec 20, 2013 at 12:34 PM, Erik Jan de Wit <[hidden email]> wrote:
So we already agreed that CouchDB has some nice features, but why wouldn’t we just use pouchdb for javascript and touchdb? I’ve just tested the javascript and it syncs with the server perfectly, even if there are some things missing we could add them because these projects are open source.

Cheers,
        Erik Jan
_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Summers Pittman
In reply to this post by Erik Jan de Wit
On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
> So we already agreed that CouchDB has some nice features, but why wouldn’t we just use pouchdb for javascript and touchdb? I’ve just tested the javascript and it syncs with the server perfectly, even if there are some things missing we could add them because these projects are open source.
Because wrapping CouchDB and friends to AeroGear with no other reason
than to create a org.jboss.sync name space is dumb.

Also the how of sync is still a bit far off. I'm still interested in the
"what" of sync. What use cases are we supporting, what features are we
supporting, what are our goals, what does a working finished sync impl
look like, etc.

>
> Cheers,
> Erik Jan
> _______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
qmx
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

qmx
Administrator
On Thu, Jan 02, 2014 at 10:13:50AM -0500, Summers Pittman wrote:
> On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
> > So we already agreed that CouchDB has some nice features, but why
> > wouldn’t we just use pouchdb for javascript and touchdb? I’ve just
> > tested the javascript and it syncs with the server perfectly, even
> > if there are some things missing we could add them because these
> > projects are open source.
> Because wrapping CouchDB and friends to AeroGear with no other reason
> than to create a org.jboss.sync name space is dumb.

can't +9001 enough

> Also the how of sync is still a bit far off. I'm still interested in the
> "what" of sync. What use cases are we supporting, what features are we
> supporting, what are our goals, what does a working finished sync impl
> look like, etc.

drafted here: https://gist.github.com/qmx/8278287

# aerogear-sync

While I was reviewing Summers' code and ideas, I realized that I really wanted
everything he did, but as a second step after we nail down the basics.

## basics?

Since we've been catering the enterprise market, this essentially means we need
to get the __boring__ stuff right first, then move over to the __shiny__ stuff,
like realtime data sync, update policies & friends.

### data model

For starters, I think that the most important thing that needs to be agreed
upon is the data model and the atomic operations around it. As previous
discussed, I really like CouchDB's datamodel -- and hate erlang ;)

`{_id:<guid>, content:<arbitrary json>, rev:<last revision>}`

#### JS

Well, it's JSON, it _Just Works_&trade;

#### Java

I didn't want to pick on Java, but since its fame forces me to it. First stab
(a courtesy of our friend Dan Bevenius):

    public interface Document {
        public String id;
        public String content;
        public String rev;
    }
   
We naturally want to kick this a notch, and use objects instead of plain strings:

    public interface Document<T, ID> {
    public ID id;
    public T content;
    public String rev;
    }
   
In this case, we can use the convention requiring that `T` is any **object
serializable to  JSON**. `ID` is a convenience shorthand since it's a
**GUID/UUID**. I think this key isn't necessarily a natural key (a surrogate
key instead).

#### Objective-C

<volunteers needed> ;)

### API levels

As soon as we have a rough data-model defined, we can start dabbling around
different API levels to be served:

(parts **I think** are potentially deliverable for a 1.0)

- level 0: explodes when there's a conflict
- level 1: semi-automatic conflict resolution via something like google's diff-match-patch
- level 2: business rules determine who wins a conflicting update (supervisor
wins over normal user)

(parts **I think** are potentially deliverable for a 2.0)

- level 3: real-time updates via diff-match-patch
- level 4: real-time updates via OT/EC

All those proposed API operations should be serializable, meaning I can
potentially keep doing changes offline then just replying them to the server
when online.

### transport

Since we know about the future-looking ideas on v2.0, it would be really nice
for us to specify a very simple/dumb JSON-based protocol for those change
messages. Something that could accomodate both the full document updates and
the OT/EC incremental bits too. I have no ideas on this, tbh.  

## boring usecases

### scenario 1

Building inspector system - we have mobile apps that store relevant info and
are bound to be accessed on places where we won't have any kind of connection,
or very poor signal.

You can have several inspectors screening the same building simultaneously.

Let's say we have Agnes and Joe are doing the fire extinguisher inspection in a
new hospital building. Technically each fire extinguisher has its own
identifier and can be an independent document. In this case we would have no
conflict happening.

Now they start finding expired fire extinguishers and start to add them to the
report. This report could potentially have two divergent lists of fire
extinguishers to be replenished/revalidated, as the building's compliance
status.

### scenario 2

Census system - we have mobile apps focused on offline data collection. We have
the previous year's info that needs to be updated on the server. The
interviewee needs to take a call, then asks the interviewer to come back later.
This results in two sets of changes for the same document, stacked together,
which should work flawlessly.

Any other ideas/comments?

--
qmx
_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev

attachment0 (539 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Luke Holmquist

On Jan 5, 2014, at 11:42 PM, Douglas Campos <[hidden email]> wrote:

> On Thu, Jan 02, 2014 at 10:13:50AM -0500, Summers Pittman wrote:
>> On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
>>> So we already agreed that CouchDB has some nice features, but why
>>> wouldn’t we just use pouchdb for javascript and touchdb? I’ve just
>>> tested the javascript and it syncs with the server perfectly, even
>>> if there are some things missing we could add them because these
>>> projects are open source.
>> Because wrapping CouchDB and friends to AeroGear with no other reason
>> than to create a org.jboss.sync name space is dumb.
>
> can't +9001 enough
>
>> Also the how of sync is still a bit far off. I'm still interested in the
>> "what" of sync. What use cases are we supporting, what features are we
>> supporting, what are our goals, what does a working finished sync impl
>> look like, etc.
>
> drafted here: https://gist.github.com/qmx/8278287
>
> # aerogear-sync
>
> While I was reviewing Summers' code and ideas, I realized that I really wanted
> everything he did, but as a second step after we nail down the basics.
>
> ## basics?
>
> Since we've been catering the enterprise market, this essentially means we need
> to get the __boring__ stuff right first, then move over to the __shiny__ stuff,
> like realtime data sync, update policies & friends.
>
> ### data model
>
> For starters, I think that the most important thing that needs to be agreed
> upon is the data model and the atomic operations around it. As previous
> discussed, I really like CouchDB's datamodel -- and hate erlang ;)
>
> `{_id:<guid>, content:<arbitrary json>, rev:<last revision>}`
>
> #### JS
>
> Well, it's JSON, it _Just Works_&trade;
>
> #### Java
>
> I didn't want to pick on Java, but since its fame forces me to it. First stab
> (a courtesy of our friend Dan Bevenius):
>
>    public interface Document {
>        public String id;
>        public String content;
>        public String rev;
>    }
>
> We naturally want to kick this a notch, and use objects instead of plain strings:
>
>    public interface Document<T, ID> {
>     public ID id;
>     public T content;
>     public String rev;
>    }
>
> In this case, we can use the convention requiring that `T` is any **object
> serializable to  JSON**. `ID` is a convenience shorthand since it's a
> **GUID/UUID**. I think this key isn't necessarily a natural key (a surrogate
> key instead).
>
> #### Objective-C
>
> <volunteers needed> ;)
>
> ### API levels
>
> As soon as we have a rough data-model defined, we can start dabbling around
> different API levels to be served:
>
> (parts **I think** are potentially deliverable for a 1.0)
>
> - level 0: explodes when there's a conflict
> - level 1: semi-automatic conflict resolution via something like google's diff-match-patch
> - level 2: business rules determine who wins a conflicting update (supervisor
> wins over normal user)
>
> (parts **I think** are potentially deliverable for a 2.0)
>
> - level 3: real-time updates via diff-match-patch
> - level 4: real-time updates via OT/EC
>
where does the more than 2 users editing something fall.

so if say 4 people are editing something offline and then come back online,  who wins that one.


> All those proposed API operations should be serializable, meaning I can
> potentially keep doing changes offline then just replying them to the server
> when online.
>
> ### transport
>
> Since we know about the future-looking ideas on v2.0, it would be really nice
> for us to specify a very simple/dumb JSON-based protocol for those change
> messages. Something that could accomodate both the full document updates and
> the OT/EC incremental bits too. I have no ideas on this, tbh.  
>
> ## boring usecases
>
> ### scenario 1
>
> Building inspector system - we have mobile apps that store relevant info and
> are bound to be accessed on places where we won't have any kind of connection,
> or very poor signal.
>
> You can have several inspectors screening the same building simultaneously.
>
> Let's say we have Agnes and Joe are doing the fire extinguisher inspection in a
> new hospital building. Technically each fire extinguisher has its own
> identifier and can be an independent document. In this case we would have no
> conflict happening.
>
> Now they start finding expired fire extinguishers and start to add them to the
> report. This report could potentially have two divergent lists of fire
> extinguishers to be replenished/revalidated, as the building's compliance
> status.
>
> ### scenario 2
>
> Census system - we have mobile apps focused on offline data collection. We have
> the previous year's info that needs to be updated on the server. The
> interviewee needs to take a call, then asks the interviewer to come back later.
> This results in two sets of changes for the same document, stacked together,
> which should work flawlessly.
>
> Any other ideas/comments?
>
> --
> qmx_______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Summers Pittman
In reply to this post by qmx
On 01/05/2014 11:42 PM, Douglas Campos wrote:
On Thu, Jan 02, 2014 at 10:13:50AM -0500, Summers Pittman wrote:
On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
So we already agreed that CouchDB has some nice features, but why
wouldn’t we just use pouchdb for javascript and touchdb? I’ve just
tested the javascript and it syncs with the server perfectly, even
if there are some things missing we could add them because these
projects are open source.
Because wrapping CouchDB and friends to AeroGear with no other reason 
than to create a org.jboss.sync name space is dumb.
can't +9001 enough

Also the how of sync is still a bit far off. I'm still interested in the 
"what" of sync. What use cases are we supporting, what features are we 
supporting, what are our goals, what does a working finished sync impl 
look like, etc.
drafted here: https://gist.github.com/qmx/8278287

# aerogear-sync

While I was reviewing Summers' code and ideas, I realized that I really wanted
everything he did, but as a second step after we nail down the basics.

## basics?

Since we've been catering the enterprise market, this essentially means we need
to get the __boring__ stuff right first, then move over to the __shiny__ stuff,
like realtime data sync, update policies & friends.

### data model

For starters, I think that the most important thing that needs to be agreed
upon is the data model and the atomic operations around it. As previous
discussed, I really like CouchDB's datamodel -- and hate erlang ;)

`{_id:<guid>, content:<arbitrary json>, rev:<last revision>}`
So the id is the id of the sync message, the content is some implementation specific message/content (could be a diff, could be a whole new object etc), and the rev is the current revision of the document on the device?

What is the scope of a revision number?  IE if there are multiple users with multiple clients on multiple devices is a revision unique per session, user, or document?


#### JS

Well, it's JSON, it _Just Works_&trade; 

#### Java

I didn't want to pick on Java, but since its fame forces me to it. First stab
(a courtesy of our friend Dan Bevenius):

    public interface Document {
        public String id;
        public String content;
        public String rev;
    }
    
We naturally want to kick this a notch, and use objects instead of plain strings:

    public interface Document<T, ID> {
    	public ID id;
    	public T content;
    	public String rev;
    }
    
In this case, we can use the convention requiring that `T` is any **object
serializable to  JSON**. `ID` is a convenience shorthand since it's a
**GUID/UUID**. I think this key isn't necessarily a natural key (a surrogate
key instead).

#### Objective-C

<volunteers needed> ;)

### API levels

As soon as we have a rough data-model defined, we can start dabbling around
different API levels to be served:

(parts **I think** are potentially deliverable for a 1.0)

- level 0: explodes when there's a conflict
- level 1: semi-automatic conflict resolution via something like google's diff-match-patch
- level 2: business rules determine who wins a conflicting update (supervisor
wins over normal user)

(parts **I think** are potentially deliverable for a 2.0)

- level 3: real-time updates via diff-match-patch
- level 4: real-time updates via OT/EC

All those proposed API operations should be serializable, meaning I can
potentially keep doing changes offline then just replying them to the server
when online.

### transport

Since we know about the future-looking ideas on v2.0, it would be really nice
for us to specify a very simple/dumb JSON-based protocol for those change
messages. Something that could accomodate both the full document updates and
the OT/EC incremental bits too. I have no ideas on this, tbh.  

## boring usecases

### scenario 1

Building inspector system - we have mobile apps that store relevant info and
are bound to be accessed on places where we won't have any kind of connection,
or very poor signal.

You can have several inspectors screening the same building simultaneously.

Let's say we have Agnes and Joe are doing the fire extinguisher inspection in a
new hospital building. Technically each fire extinguisher has its own
identifier and can be an independent document. In this case we would have no
conflict happening.

Now they start finding expired fire extinguishers and start to add them to the
report. This report could potentially have two divergent lists of fire
extinguishers to be replenished/revalidated, as the building's compliance
status.

### scenario 2

Census system - we have mobile apps focused on offline data collection. We have
the previous year's info that needs to be updated on the server. The
interviewee needs to take a call, then asks the interviewer to come back later.
This results in two sets of changes for the same document, stacked together,
which should work flawlessly.

Any other ideas/comments?
Does the developer work with their objects and AeroGear create/manage the Document objects behind the scenes or does the developer have to manage the Document objects themselves?

Is the Document object a change instance or a full implementation of an instance of the document? 

What does the server request/response lifecycle look like?

How are updates from outside of the client handled?

Are we including a server implementation ala unified push or are we only focusing on a spec that a server will implement?


      

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Summers Pittman
In reply to this post by Luke Holmquist
On 01/06/2014 09:29 AM, Lucas Holmquist wrote:

> On Jan 5, 2014, at 11:42 PM, Douglas Campos <[hidden email]> wrote:
>
>> On Thu, Jan 02, 2014 at 10:13:50AM -0500, Summers Pittman wrote:
>>> On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
>>>> So we already agreed that CouchDB has some nice features, but why
>>>> wouldn’t we just use pouchdb for javascript and touchdb? I’ve just
>>>> tested the javascript and it syncs with the server perfectly, even
>>>> if there are some things missing we could add them because these
>>>> projects are open source.
>>> Because wrapping CouchDB and friends to AeroGear with no other reason
>>> than to create a org.jboss.sync name space is dumb.
>> can't +9001 enough
>>
>>> Also the how of sync is still a bit far off. I'm still interested in the
>>> "what" of sync. What use cases are we supporting, what features are we
>>> supporting, what are our goals, what does a working finished sync impl
>>> look like, etc.
>> drafted here: https://gist.github.com/qmx/8278287
>>
>> # aerogear-sync
>>
>> While I was reviewing Summers' code and ideas, I realized that I really wanted
>> everything he did, but as a second step after we nail down the basics.
>>
>> ## basics?
>>
>> Since we've been catering the enterprise market, this essentially means we need
>> to get the __boring__ stuff right first, then move over to the __shiny__ stuff,
>> like realtime data sync, update policies & friends.
>>
>> ### data model
>>
>> For starters, I think that the most important thing that needs to be agreed
>> upon is the data model and the atomic operations around it. As previous
>> discussed, I really like CouchDB's datamodel -- and hate erlang ;)
>>
>> `{_id:<guid>, content:<arbitrary json>, rev:<last revision>}`
>>
>> #### JS
>>
>> Well, it's JSON, it _Just Works_&trade;
>>
>> #### Java
>>
>> I didn't want to pick on Java, but since its fame forces me to it. First stab
>> (a courtesy of our friend Dan Bevenius):
>>
>>     public interface Document {
>>         public String id;
>>         public String content;
>>         public String rev;
>>     }
>>
>> We naturally want to kick this a notch, and use objects instead of plain strings:
>>
>>     public interface Document<T, ID> {
>>     public ID id;
>>     public T content;
>>     public String rev;
>>     }
>>
>> In this case, we can use the convention requiring that `T` is any **object
>> serializable to  JSON**. `ID` is a convenience shorthand since it's a
>> **GUID/UUID**. I think this key isn't necessarily a natural key (a surrogate
>> key instead).
>>
>> #### Objective-C
>>
>> <volunteers needed> ;)
>>
>> ### API levels
>>
>> As soon as we have a rough data-model defined, we can start dabbling around
>> different API levels to be served:
>>
>> (parts **I think** are potentially deliverable for a 1.0)
>>
>> - level 0: explodes when there's a conflict
>> - level 1: semi-automatic conflict resolution via something like google's diff-match-patch
>> - level 2: business rules determine who wins a conflicting update (supervisor
>> wins over normal user)
>>
>> (parts **I think** are potentially deliverable for a 2.0)
>>
>> - level 3: real-time updates via diff-match-patch
>> - level 4: real-time updates via OT/EC
>>
> where does the more than 2 users editing something fall.
>
> so if say 4 people are editing something offline and then come back online,  who wins that one.
I think that would be level 2.  Either the developer would write a
business rule or choose a provided conflict resolution strategy object.

I think AbstractConflictResolutionStrategyFactory is too awesome of a
class name to miss out on now :)

>
>
>> All those proposed API operations should be serializable, meaning I can
>> potentially keep doing changes offline then just replying them to the server
>> when online.
>>
>> ### transport
>>
>> Since we know about the future-looking ideas on v2.0, it would be really nice
>> for us to specify a very simple/dumb JSON-based protocol for those change
>> messages. Something that could accomodate both the full document updates and
>> the OT/EC incremental bits too. I have no ideas on this, tbh.
>>
>> ## boring usecases
>>
>> ### scenario 1
>>
>> Building inspector system - we have mobile apps that store relevant info and
>> are bound to be accessed on places where we won't have any kind of connection,
>> or very poor signal.
>>
>> You can have several inspectors screening the same building simultaneously.
>>
>> Let's say we have Agnes and Joe are doing the fire extinguisher inspection in a
>> new hospital building. Technically each fire extinguisher has its own
>> identifier and can be an independent document. In this case we would have no
>> conflict happening.
>>
>> Now they start finding expired fire extinguishers and start to add them to the
>> report. This report could potentially have two divergent lists of fire
>> extinguishers to be replenished/revalidated, as the building's compliance
>> status.
>>
>> ### scenario 2
>>
>> Census system - we have mobile apps focused on offline data collection. We have
>> the previous year's info that needs to be updated on the server. The
>> interviewee needs to take a call, then asks the interviewer to come back later.
>> This results in two sets of changes for the same document, stacked together,
>> which should work flawlessly.
>>
>> Any other ideas/comments?
>>
>> --
>> qmx_______________________________________________
>> aerogear-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/aerogear-dev
>
> _______________________________________________
> aerogear-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/aerogear-dev

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Summers Pittman
In reply to this post by qmx
On 01/05/2014 11:42 PM, Douglas Campos wrote:
On Thu, Jan 02, 2014 at 10:13:50AM -0500, Summers Pittman wrote:
On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
So we already agreed that CouchDB has some nice features, but why
wouldn’t we just use pouchdb for javascript and touchdb? I’ve just
tested the javascript and it syncs with the server perfectly, even
if there are some things missing we could add them because these
projects are open source.
Because wrapping CouchDB and friends to AeroGear with no other reason 
than to create a org.jboss.sync name space is dumb.
can't +9001 enough

Also the how of sync is still a bit far off. I'm still interested in the 
"what" of sync. What use cases are we supporting, what features are we 
supporting, what are our goals, what does a working finished sync impl 
look like, etc.
drafted here: https://gist.github.com/qmx/8278287

# aerogear-sync

While I was reviewing Summers' code and ideas, I realized that I really wanted
everything he did, but as a second step after we nail down the basics.

## basics?

Since we've been catering the enterprise market, this essentially means we need
to get the __boring__ stuff right first, then move over to the __shiny__ stuff,
like realtime data sync, update policies & friends.

### data model

For starters, I think that the most important thing that needs to be agreed
upon is the data model and the atomic operations around it. As previous
discussed, I really like CouchDB's datamodel -- and hate erlang ;)

`{_id:<guid>, content:<arbitrary json>, rev:<last revision>}`

#### JS

Well, it's JSON, it _Just Works_&trade; 

#### Java

I didn't want to pick on Java, but since its fame forces me to it. First stab
(a courtesy of our friend Dan Bevenius):

    public interface Document {
        public String id;
        public String content;
        public String rev;
    }
    
We naturally want to kick this a notch, and use objects instead of plain strings:

    public interface Document<T, ID> {
    	public ID id;
    	public T content;
    	public String rev;
    }
    
In this case, we can use the convention requiring that `T` is any **object
serializable to  JSON**. `ID` is a convenience shorthand since it's a
**GUID/UUID**. I think this key isn't necessarily a natural key (a surrogate
key instead).

#### Objective-C

<volunteers needed> ;)

### API levels

As soon as we have a rough data-model defined, we can start dabbling around
different API levels to be served:

(parts **I think** are potentially deliverable for a 1.0)

- level 0: explodes when there's a conflict
- level 1: semi-automatic conflict resolution via something like google's diff-match-patch
- level 2: business rules determine who wins a conflicting update (supervisor
wins over normal user)

(parts **I think** are potentially deliverable for a 2.0)

- level 3: real-time updates via diff-match-patch
- level 4: real-time updates via OT/EC

All those proposed API operations should be serializable, meaning I can
potentially keep doing changes offline then just replying them to the server
when online.

### transport

Since we know about the future-looking ideas on v2.0, it would be really nice
for us to specify a very simple/dumb JSON-based protocol for those change
messages. Something that could accomodate both the full document updates and
the OT/EC incremental bits too. I have no ideas on this, tbh.  

## boring usecases

### scenario 1

Building inspector system - we have mobile apps that store relevant info and
are bound to be accessed on places where we won't have any kind of connection,
or very poor signal.

You can have several inspectors screening the same building simultaneously.

Let's say we have Agnes and Joe are doing the fire extinguisher inspection in a
new hospital building. Technically each fire extinguisher has its own
identifier and can be an independent document. In this case we would have no
conflict happening.

Now they start finding expired fire extinguishers and start to add them to the
report. This report could potentially have two divergent lists of fire
extinguishers to be replenished/revalidated, as the building's compliance
status.

### scenario 2

Census system - we have mobile apps focused on offline data collection. We have
the previous year's info that needs to be updated on the server. The
interviewee needs to take a call, then asks the interviewer to come back later.
This results in two sets of changes for the same document, stacked together,
which should work flawlessly.

Any other ideas/comments?

I'll add now :)

## Transactions

These are the most basic parts of sync I can think of that our system should be able to do/manage.  Our internal representation of the client documents and collections should make implementing this automatically and without user intervention as simple as possible

* Detect Change
    When a user changes her local data, the system should note the change and generate a sync message to send to the server.  This can be done automatically or manually but SHOULD be done automatically.


* Send update

    When a sync message is ready to be sent, and the system allows for it to be sent (network available, not in blackout window from exponential backoff, etc) then sync message should be sent.  This SHOULD be done automatically. 


* Receive Update
    When a client updates it data and successfully syncs to the remote server, the remote server will notify all of the relevant clients.  The client must automatically and without user intervention receive this update and either act on it or store it for later processing.

* Apply Update
    Once a client application has an update message from the server, it can apply the message correctly to the data on it.  This should be done automatically as part of receiving the update, but it may be done manually or may be delayed and automatically executed later.

* Detect Conflict
    When applying an update fails, the system must detect this.  The system will provide state to the application and/or the user to handle the conflict.  The user MUST NOT have to check for conflicts on her own.

* Resolve Conflict
    There must be a mechanism for resolving a conflict.  The CAN be done automatically using default resolvers provided by AeroGear, by using a resolver provided by the developer/user, or by the app user selecting the correct merge.  This will probably generate a new sync message.






_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Summers Pittman
In reply to this post by qmx
On 01/05/2014 11:42 PM, Douglas Campos wrote:
On Thu, Jan 02, 2014 at 10:13:50AM -0500, Summers Pittman wrote:
On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
So we already agreed that CouchDB has some nice features, but why
wouldn’t we just use pouchdb for javascript and touchdb? I’ve just
tested the javascript and it syncs with the server perfectly, even
if there are some things missing we could add them because these
projects are open source.
Because wrapping CouchDB and friends to AeroGear with no other reason 
than to create a org.jboss.sync name space is dumb.
can't +9001 enough

Also the how of sync is still a bit far off. I'm still interested in the 
"what" of sync. What use cases are we supporting, what features are we 
supporting, what are our goals, what does a working finished sync impl 
look like, etc.
drafted here: https://gist.github.com/qmx/8278287

# aerogear-sync

While I was reviewing Summers' code and ideas, I realized that I really wanted
everything he did, but as a second step after we nail down the basics.

## basics?

Since we've been catering the enterprise market, this essentially means we need
to get the __boring__ stuff right first, then move over to the __shiny__ stuff,
like realtime data sync, update policies & friends.

### data model

For starters, I think that the most important thing that needs to be agreed
upon is the data model and the atomic operations around it. As previous
discussed, I really like CouchDB's datamodel -- and hate erlang ;)

`{_id:<guid>, content:<arbitrary json>, rev:<last revision>}`

#### JS

Well, it's JSON, it _Just Works_&trade; 

#### Java

I didn't want to pick on Java, but since its fame forces me to it. First stab
(a courtesy of our friend Dan Bevenius):

    public interface Document {
        public String id;
        public String content;
        public String rev;
    }
    
We naturally want to kick this a notch, and use objects instead of plain strings:

    public interface Document<T, ID> {
    	public ID id;
    	public T content;
    	public String rev;
    }
    
In this case, we can use the convention requiring that `T` is any **object
serializable to  JSON**. `ID` is a convenience shorthand since it's a
**GUID/UUID**. I think this key isn't necessarily a natural key (a surrogate
key instead).

#### Objective-C

<volunteers needed> ;)

### API levels

As soon as we have a rough data-model defined, we can start dabbling around
different API levels to be served:

(parts **I think** are potentially deliverable for a 1.0)

- level 0: explodes when there's a conflict
- level 1: semi-automatic conflict resolution via something like google's diff-match-patch
- level 2: business rules determine who wins a conflicting update (supervisor
wins over normal user)

(parts **I think** are potentially deliverable for a 2.0)

- level 3: real-time updates via diff-match-patch
- level 4: real-time updates via OT/EC

All those proposed API operations should be serializable, meaning I can
potentially keep doing changes offline then just replying them to the server
when online.

### transport

Since we know about the future-looking ideas on v2.0, it would be really nice
for us to specify a very simple/dumb JSON-based protocol for those change
messages. Something that could accomodate both the full document updates and
the OT/EC incremental bits too. I have no ideas on this, tbh.  

A good transport will include the change to sync (this can be a diff or a whole object), the revision on the client, and a checksum so the server can be sure that the object was updated correctly.


## boring usecases

### scenario 1

Building inspector system - we have mobile apps that store relevant info and
are bound to be accessed on places where we won't have any kind of connection,
or very poor signal.

You can have several inspectors screening the same building simultaneously.

Let's say we have Agnes and Joe are doing the fire extinguisher inspection in a
new hospital building. Technically each fire extinguisher has its own
identifier and can be an independent document. In this case we would have no
conflict happening.

Now they start finding expired fire extinguishers and start to add them to the
report. This report could potentially have two divergent lists of fire
extinguishers to be replenished/revalidated, as the building's compliance
status.

### scenario 2

Census system - we have mobile apps focused on offline data collection. We have
the previous year's info that needs to be updated on the server. The
interviewee needs to take a call, then asks the interviewer to come back later.
This results in two sets of changes for the same document, stacked together,
which should work flawlessly.

Any other ideas/comments?



_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] sync

Luke Holmquist
In reply to this post by Summers Pittman
i might be getting ahead of things,  but i put together an initial thought of what i think the JS lib will sort of look like for sync


i know there are somethings i still need to add,  but i needed to get the thoughts out of my head

On Jan 6, 2014, at 11:30 AM, Summers Pittman <[hidden email]> wrote:

On 01/05/2014 11:42 PM, Douglas Campos wrote:
On Thu, Jan 02, 2014 at 10:13:50AM -0500, Summers Pittman wrote:
On 12/20/2013 06:34 AM, Erik Jan de Wit wrote:
So we already agreed that CouchDB has some nice features, but why
wouldn’t we just use pouchdb for javascript and touchdb? I’ve just
tested the javascript and it syncs with the server perfectly, even
if there are some things missing we could add them because these
projects are open source.
Because wrapping CouchDB and friends to AeroGear with no other reason 
than to create a org.jboss.sync name space is dumb.
can't +9001 enough

Also the how of sync is still a bit far off. I'm still interested in the 
"what" of sync. What use cases are we supporting, what features are we 
supporting, what are our goals, what does a working finished sync impl 
look like, etc.
drafted here: https://gist.github.com/qmx/8278287

# aerogear-sync

While I was reviewing Summers' code and ideas, I realized that I really wanted
everything he did, but as a second step after we nail down the basics.

## basics?

Since we've been catering the enterprise market, this essentially means we need
to get the __boring__ stuff right first, then move over to the __shiny__ stuff,
like realtime data sync, update policies & friends.
### data model

For starters, I think that the most important thing that needs to be agreed
upon is the data model and the atomic operations around it. As previous
discussed, I really like CouchDB's datamodel -- and hate erlang ;)

`{_id:<guid>, content:<arbitrary json>, rev:<last revision>}`

#### JS

Well, it's JSON, it _Just Works_&trade; 

#### Java

I didn't want to pick on Java, but since its fame forces me to it. First stab
(a courtesy of our friend Dan Bevenius):

    public interface Document {
        public String id;
        public String content;
        public String rev;
    }
    
We naturally want to kick this a notch, and use objects instead of plain strings:

    public interface Document<T, ID> {
    	public ID id;
    	public T content;
    	public String rev;
    }
    
In this case, we can use the convention requiring that `T` is any **object
serializable to  JSON**. `ID` is a convenience shorthand since it's a
**GUID/UUID**. I think this key isn't necessarily a natural key (a surrogate
key instead).

#### Objective-C

<volunteers needed> ;)

### API levels

As soon as we have a rough data-model defined, we can start dabbling around
different API levels to be served:

(parts **I think** are potentially deliverable for a 1.0)

- level 0: explodes when there's a conflict
- level 1: semi-automatic conflict resolution via something like google's diff-match-patch
- level 2: business rules determine who wins a conflicting update (supervisor
wins over normal user)

(parts **I think** are potentially deliverable for a 2.0)

- level 3: real-time updates via diff-match-patch
- level 4: real-time updates via OT/EC

All those proposed API operations should be serializable, meaning I can
potentially keep doing changes offline then just replying them to the server
when online.

### transport

Since we know about the future-looking ideas on v2.0, it would be really nice
for us to specify a very simple/dumb JSON-based protocol for those change
messages. Something that could accomodate both the full document updates and
the OT/EC incremental bits too. I have no ideas on this, tbh.  

## boring usecases

### scenario 1

Building inspector system - we have mobile apps that store relevant info and
are bound to be accessed on places where we won't have any kind of connection,
or very poor signal.

You can have several inspectors screening the same building simultaneously.

Let's say we have Agnes and Joe are doing the fire extinguisher inspection in a
new hospital building. Technically each fire extinguisher has its own
identifier and can be an independent document. In this case we would have no
conflict happening.

Now they start finding expired fire extinguishers and start to add them to the
report. This report could potentially have two divergent lists of fire
extinguishers to be replenished/revalidated, as the building's compliance
status.

### scenario 2

Census system - we have mobile apps focused on offline data collection. We have
the previous year's info that needs to be updated on the server. The
interviewee needs to take a call, then asks the interviewer to come back later.
This results in two sets of changes for the same document, stacked together,
which should work flawlessly.

Any other ideas/comments?

I'll add now :)

## Transactions

These are the most basic parts of sync I can think of that our system should be able to do/manage.  Our internal representation of the client documents and collections should make implementing this automatically and without user intervention as simple as possible

* Detect Change
    When a user changes her local data, the system should note the change and generate a sync message to send to the server.  This can be done automatically or manually but SHOULD be done automatically.


* Send update

    When a sync message is ready to be sent, and the system allows for it to be sent (network available, not in blackout window from exponential backoff, etc) then sync message should be sent.  This SHOULD be done automatically. 


* Receive Update
    When a client updates it data and successfully syncs to the remote server, the remote server will notify all of the relevant clients.  The client must automatically and without user intervention receive this update and either act on it or store it for later processing.

* Apply Update
    Once a client application has an update message from the server, it can apply the message correctly to the data on it.  This should be done automatically as part of receiving the update, but it may be done manually or may be delayed and automatically executed later.

* Detect Conflict
    When applying an update fails, the system must detect this.  The system will provide state to the application and/or the user to handle the conflict.  The user MUST NOT have to check for conflicts on her own.

* Resolve Conflict
    There must be a mechanism for resolving a conflict.  The CAN be done automatically using default resolvers provided by AeroGear, by using a resolver provided by the developer/user, or by the app user selecting the correct merge.  This will probably generate a new sync message.




      

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev