[aerogear-dev] aerogear-js CouchDB data-manager adapter

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[aerogear-dev] aerogear-js CouchDB data-manager adapter

tolis emmanouilidis
Hi,

I have started working on creating a CouchDB data-manager adapter which uses the CouchDB REST API. The open-create database, read and save operations are implemented [1] and tested both with encryption settings and without encryption settings. If you'd like to give it a try, make sure that you have enabled CORS on CouchDB side.

The remaining functions that needs to be implemented are remove and filter. In addition, in order to keep consistent the data-manager API, we should provide a way to remove all the documents in a CouchDB database-store.

a) Removing all documents in a CouchDB database

I'm thinking of fetching all the docs, using _all_docs endpoint which returns _id and _rev (revision), mark them as _deleted and use the bulk insert/update API to delete them. Not sure if this is a proper solution since there might be a huge number of documents.

Another option would be to delete the database-store. This seems to be the worst solution since the documents history will be permanently lost.

Any ideas?

b) Filtering

AFAIK, in order to perform filtering in a CouchDB database, a design document and an appropriate view has to be created. My thought is that the AGJS adapter should not create the view, instead we should assume that the user has created the appopriate views and provide a setting in AGJS where the view endpoint URl can be set. 

Any ideas about filtering and the proper way to be implemented?

Thanks,
Tolis


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] aerogear-js CouchDB data-manager adapter

Lukas Fryc
Hey Tolis,

great job with the adapter!


a) remove()

since there are obviously many ways how to achieve "delete all docs",
I believe it's up to user to choose the way he wants the docs deleted
i.e. it could be up to Data Manager configuration whether data will be deleted with or without a history loss (aka wipe out). wdyt?


b) Filtering

It would be pretty overwhelming for a user to create a view per particular use of the filter() method, since it can have pretty arbitrary form.
We are also able to create temporary views, but that requires you to perform one additional POST request and it is costly.

Are we able to come up with a common view definition that would cover all the filtering capabilities - i.e. generic aerogear-filter view?
Something that user would define once and all cases would be covered.
I have not practically played with CouchDB, but according the docs it could be somehow possible.



Btw as I think about it, there might be lack of function for limiting what data to transfer. 
i.e. Filtering API allows you to select just particular docs, but it does not help you to avoid what will be transferred.

All the Data Managers so far are local ones, CouchDB is a first one that actually transfers data from the wire.


Cheers,

~ Lukas


On Mon, May 26, 2014 at 8:40 AM, tolis emmanouilidis <[hidden email]> wrote:
Hi,

I have started working on creating a CouchDB data-manager adapter which uses the CouchDB REST API. The open-create database, read and save operations are implemented [1] and tested both with encryption settings and without encryption settings. If you'd like to give it a try, make sure that you have enabled CORS on CouchDB side.

The remaining functions that needs to be implemented are remove and filter. In addition, in order to keep consistent the data-manager API, we should provide a way to remove all the documents in a CouchDB database-store.

a) Removing all documents in a CouchDB database

I'm thinking of fetching all the docs, using _all_docs endpoint which returns _id and _rev (revision), mark them as _deleted and use the bulk insert/update API to delete them. Not sure if this is a proper solution since there might be a huge number of documents.

Another option would be to delete the database-store. This seems to be the worst solution since the documents history will be permanently lost.

Any ideas?

b) Filtering

AFAIK, in order to perform filtering in a CouchDB database, a design document and an appropriate view has to be created. My thought is that the AGJS adapter should not create the view, instead we should assume that the user has created the appopriate views and provide a setting in AGJS where the view endpoint URl can be set. 

Any ideas about filtering and the proper way to be implemented?

Thanks,
Tolis


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] aerogear-js CouchDB data-manager adapter

Luke Holmquist
Cool Stuff,  i will take a look

On May 26, 2014, at 7:55 AM, Lukáš Fryč <[hidden email]> wrote:

Hey Tolis,

great job with the adapter!


a) remove()

since there are obviously many ways how to achieve "delete all docs",
I believe it's up to user to choose the way he wants the docs deleted
i.e. it could be up to Data Manager configuration whether data will be deleted with or without a history loss (aka wipe out). wdyt?


b) Filtering

It would be pretty overwhelming for a user to create a view per particular use of the filter() method, since it can have pretty arbitrary form.
We are also able to create temporary views, but that requires you to perform one additional POST request and it is costly.

Are we able to come up with a common view definition that would cover all the filtering capabilities - i.e. generic aerogear-filter view?
Something that user would define once and all cases would be covered.
I have not practically played with CouchDB, but according the docs it could be somehow possible.



Btw as I think about it, there might be lack of function for limiting what data to transfer. 
i.e. Filtering API allows you to select just particular docs, but it does not help you to avoid what will be transferred.

All the Data Managers so far are local ones, CouchDB is a first one that actually transfers data from the wire.

This is a concern i had when created the JIRA,   This “adapter” might be more appropriate for DataSync( or whatever we are calling it ).  I know we were looking at couchDB as a possible backend.

I’m wondering if we should hold off for now,  and just keep DataManager client side storage only for the moment.  I think the fallback functionality will be non-trivial  




Cheers,

~ Lukas


On Mon, May 26, 2014 at 8:40 AM, tolis emmanouilidis <[hidden email]> wrote:
Hi,

I have started working on creating a CouchDB data-manager adapter which uses the CouchDB REST API. The open-create database, read and save operations are implemented [1] and tested both with encryption settings and without encryption settings. If you'd like to give it a try, make sure that you have enabled CORS on CouchDB side.

The remaining functions that needs to be implemented are remove and filter. In addition, in order to keep consistent the data-manager API, we should provide a way to remove all the documents in a CouchDB database-store.

a) Removing all documents in a CouchDB database

I'm thinking of fetching all the docs, using _all_docs endpoint which returns _id and _rev (revision), mark them as _deleted and use the bulk insert/update API to delete them. Not sure if this is a proper solution since there might be a huge number of documents.

Another option would be to delete the database-store. This seems to be the worst solution since the documents history will be permanently lost.

Any ideas?

b) Filtering

AFAIK, in order to perform filtering in a CouchDB database, a design document and an appropriate view has to be created. My thought is that the AGJS adapter should not create the view, instead we should assume that the user has created the appopriate views and provide a setting in AGJS where the view endpoint URl can be set. 

Any ideas about filtering and the proper way to be implemented?

Thanks,
Tolis


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] aerogear-js CouchDB data-manager adapter

tolis emmanouilidis
Thank you both.

I agree to freeze the adapter's development until datasync is discussed.

comments inline

2014-05-27 16:12 GMT+03:00 Lucas Holmquist <[hidden email]>:
Cool Stuff,  i will take a look

On May 26, 2014, at 7:55 AM, Lukáš Fryč <[hidden email]> wrote:

Hey Tolis,

great job with the adapter!


a) remove()

since there are obviously many ways how to achieve "delete all docs",
I believe it's up to user to choose the way he wants the docs deleted
i.e. it could be up to Data Manager configuration whether data will be deleted with or without a history loss (aka wipe out). wdyt?

+1 

investigation is needed to find out if there are any best practices
 

b) Filtering

It would be pretty overwhelming for a user to create a view per particular use of the filter() method, since it can have pretty arbitrary form.
We are also able to create temporary views, but that requires you to perform one additional POST request and it is costly.

I think it's not overwelming for a user to create the view manually. CouchDB filtering is based on the key (simple or complex). IMO the ability to search/filter different fields (which are not part of a complex key) and create a view in each request, has no meaning in CouchDB. If a user has a such requirement and data-queries are changing very often, then he should consider using a different NoSQL db. CouchDB serves specific purposes and a possible AeroGear JS CouchDB adapter should not offer functionality which CouchDB is not designed to serve.  

Temporary views are not a solution, for sure. They are one-off queries, meaning that they are computed in each call (expensive and slow). CouchDB docs suggest to use them for development purposes.
 
Are we able to come up with a common view definition that would cover all the filtering capabilities - i.e. generic aerogear-filter view?
Something that user would define once and all cases would be covered.
I have not practically played with CouchDB, but according the docs it could be somehow possible.



Btw as I think about it, there might be lack of function for limiting what data to transfer. 
i.e. Filtering API allows you to select just particular docs, but it does not help you to avoid what will be transferred.

All the Data Managers so far are local ones, CouchDB is a first one that actually transfers data from the wire.

This is a concern i had when created the JIRA,   This “adapter” might be more appropriate for DataSync( or whatever we are calling it ).  I know we were looking at couchDB as a possible backend.

I’m wondering if we should hold off for now,  and just keep DataManager client side storage only for the moment.  I think the fallback functionality will be non-trivial  

agreed
 

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev
Reply | Threaded
Open this post in threaded view
|

Re: [aerogear-dev] aerogear-js CouchDB data-manager adapter

Lukas Fryc
He guys,

resurrecting this topic,

I wonder whether it would make sense to do PouchDB adapter,
which is client-side API-compatible version of CouchDB,
that can be synchronized against CouchDB (due to REST API compatibility).

Wdyt?

~ Lukas


On Wed, May 28, 2014 at 9:19 AM, tolis emmanouilidis <[hidden email]> wrote:
Thank you both.

I agree to freeze the adapter's development until datasync is discussed.

comments inline

2014-05-27 16:12 GMT+03:00 Lucas Holmquist <[hidden email]>:

Cool Stuff,  i will take a look

On May 26, 2014, at 7:55 AM, Lukáš Fryč <[hidden email]> wrote:

Hey Tolis,

great job with the adapter!


a) remove()

since there are obviously many ways how to achieve "delete all docs",
I believe it's up to user to choose the way he wants the docs deleted
i.e. it could be up to Data Manager configuration whether data will be deleted with or without a history loss (aka wipe out). wdyt?

+1 

investigation is needed to find out if there are any best practices
 

b) Filtering

It would be pretty overwhelming for a user to create a view per particular use of the filter() method, since it can have pretty arbitrary form.
We are also able to create temporary views, but that requires you to perform one additional POST request and it is costly.

I think it's not overwelming for a user to create the view manually. CouchDB filtering is based on the key (simple or complex). IMO the ability to search/filter different fields (which are not part of a complex key) and create a view in each request, has no meaning in CouchDB. If a user has a such requirement and data-queries are changing very often, then he should consider using a different NoSQL db. CouchDB serves specific purposes and a possible AeroGear JS CouchDB adapter should not offer functionality which CouchDB is not designed to serve.  

Temporary views are not a solution, for sure. They are one-off queries, meaning that they are computed in each call (expensive and slow). CouchDB docs suggest to use them for development purposes.
 
Are we able to come up with a common view definition that would cover all the filtering capabilities - i.e. generic aerogear-filter view?
Something that user would define once and all cases would be covered.
I have not practically played with CouchDB, but according the docs it could be somehow possible.



Btw as I think about it, there might be lack of function for limiting what data to transfer. 
i.e. Filtering API allows you to select just particular docs, but it does not help you to avoid what will be transferred.

All the Data Managers so far are local ones, CouchDB is a first one that actually transfers data from the wire.

This is a concern i had when created the JIRA,   This “adapter” might be more appropriate for DataSync( or whatever we are calling it ).  I know we were looking at couchDB as a possible backend.

I’m wondering if we should hold off for now,  and just keep DataManager client side storage only for the moment.  I think the fallback functionality will be non-trivial  

agreed
 

_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev


_______________________________________________
aerogear-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-dev