write 2 different files (question)

[gustavoca]

Hello, is it possible that with 1 pump read from the source once, and write to 2 output files??, the first one will be a .csv file and the second one will be an excel...

this way i wouldnt have to read again the source.. if that makes sense..

Thanks!.

[novaki]

Hi, you can do it with three pumps:

{ mixin: { CsvWriterMixin, ExcelWriterMixin }, Pump } = require 'datapumps'
Promise = require 'bluebird'

(sourceReader = new Pump())
  .buffers
    csvReport: sourceReader.createBuffer()
    xlsxReport: sourceReader.createBuffer()
  .process (item) ->
    Promise.all [
      @buffer('csvReport').writeAsync item
      @buffer('xlsxReport').writeAsync item
    ]

(csvWriter = new Pump())
  .from sourceReader.buffer 'csvReport'
  ...

(xlsxWriter = new Pump())
  .from sourceReader.buffer 'xlsxReport'
  ...

Promise.all [ sourceReader.run(), csvWriter.run(), xlsxWriter.run() ]

In the sourceReader pump, I've created two output buffers, and wrote the same data to both of them in process. Then I just created two other pumps reading from those buffers.

[gustavoca]

Ohhh, so I'm guessing that in sourceReader pump I specify the "from" to let's say a mysql db, correct?
Thanks for the clarification

[novaki]

Yes, that's right.

[gustavoca]

Hey, so ive made my example :).

http://pastebin.com/vDVnV9EP

but when i run it some items goes to one buffer while others go to the other..
so i end up with some items in the csv file and the rest in the xlsx file :/

is there something im missing there?

[novaki]

Hi, I was debugging the code for an hour after I've found out, that both csvWriter and xlsxWriter reads from csvReport buffer :). Just change csvReport to xlsxReport in line 44 and it should be ok.

[gustavoca]

Wow thats embarrasing, sorry for wasting your time..

but thank you!

[novaki]

No problem, datapumps should have reported this.

Your welcome.

[gustavoca]

Hello, me again.. so im trying to filter data inside the process method using a single pump

http://pastebin.com/U9H2SxqP

like that, but im getting
Error in pump (root): Error: .process() did not return a Promise
Unhandled rejection Error: Pumping failed. See .errorBuffer() contents for error messages

isnt pump.writeRow overrides the need of a promise?

[novaki]

No it does not. In fact, pump.writeRow() returns a promise, which can be returned in process.

My recommendation is to make orderDtmfs return a promise (its uses a cb now, I assume), and then you could do something like this:

.process(function(account) {
  return orderDtmfs(account)
    .then(function(dtmfs) {
      return pump.writeRow([ account.cdr_id, account.account_id, dtmfs ]);
    });