Feature request: write data to db

[jmarca]

Nice library; it works pretty well with very few issues.

One feature I am missing is the ability to write data to the db. It seems that this library can only execute queries against existing data.

My approach in perl is to set up a copy statement, then write data, then execute the statement. I would think this approach would work well in node. Set up a copy statement that returns a stream, load chunks of data into the stream, then trigger an end event.

(I know a patch with this functionality would be better than a feature request, but at the moment I can't do more than this, sorry.)

[brianc]

Thanks for your feedback! I do admit I didnt implement the "copy" functionality on the connection object since I didnt really understand exactly how it would be used. If you could provide an example use case in JavaScript, perl, or even pseudo-code of what you're trying to do, I'll get to work on the feature. I'm on vacation until Sunday so it might take a few days to knock it out, but if you can get me the example by tomorrow I might be able to take a look at it while I'm by the pool. ;)

[jmarca]

While I would never want anyone to work while on vacation...for when you get back, here is a snippet of perl code I am using.

I've got a CSV file I am parsing, reasonably well formatted with a header row and then lots of data. Copy is orders of magnitude faster than reading lines and putting them each into the db, so I must use it in cases like this where I have lots of data to process.

The best source of info on this is probably at the CPAN: http://search.cpan.org/~turnstep/DBD-Pg-2.17.2/Pg.pm#COPY_support. If you grok perl, you can look at the source code to see what they're doing, but if you just want a use case:

  $dbh->do(
'COPY imputed.vds_hourly (tod,day,ts,vds_id,lane,heavyheavy,not_heavyheavy,n,o)  FROM STDIN with  NULL AS  \'NA\' CSV HEADER'
);

  while ( my $line = $fh->getline() ) {
      $dbh->pg_putcopydata($line);
  }

  $dbh->pg_putcopyend();

And the data is in the db.

[brianc]

I found the related client/server protocol messages related to CopyIn and CopyOut operations while I was away, but didn't get any time to hack on it. As soon as things settle down I'll start to work on this.

[brianc]

I'm waiting for libuv to be fully integrated into node so I can use the C++ buffer access for the native copy-in/copy-out bindings. The pure javascript version is fairly trivial.

[jmarca]

Any progress on the copy statement implementation?

[brianc]

Not yet. If you have some spare time and you'd like, contact me via gchat
and maybe we can get you working on the implementation.

On Thu, Jul 26, 2012 at 2:17 PM, James Marca <
[email protected]

wrote:

Any progress on the copy statement implementation?

---

Reply to this email directly or view it on GitHub:
#7 (comment)

[jmarca]

I started looking at it...lordy my C is rusty. I may pursue this slowly as time permits, but my implementation will suck.

I'll post/send pull requests if I get a working implementation.

[brianc]

Try working on the javascript implementation. I'm available to give you
pointers or review a fork. If you get the javascript implementation
working with tests that would be preferable and most likely a bit easier.

On Thu, Jul 26, 2012 at 3:32 PM, James Marca <
[email protected]

wrote:

I started looking at it...lordy my C is rusty. I may pursue this slowly
as time permits, but my implementation will suck.

I'll post/send pull requests if I get a working implementation.

---

Reply to this email directly or view it on GitHub:
#7 (comment)

[brianc]

If you get stuck I will work on it - no worries - I need to do SSL support
first though. And there's some work on connection pooling, and a long
outstanding pull request for query canceling that needs some of my
attention.

On Fri, Jul 27, 2012 at 10:34 AM, Brian Carlson
[email protected]:

Try working on the javascript implementation. I'm available to give you
pointers or review a fork. If you get the javascript implementation
working with tests that would be preferable and most likely a bit easier.

On Thu, Jul 26, 2012 at 3:32 PM, James Marca <
[email protected]

wrote:

I started looking at it...lordy my C is rusty. I may pursue this slowly
as time permits, but my implementation will suck.

I'll post/send pull requests if I get a working implementation.

---

Reply to this email directly or view it on GitHub:
#7 (comment)

[brianc]

This finally got closed! #232