Don't send over useless verbose log messages.

When in :data logging mode we log the whole data set as we read then
write it, which is quite a lot of data. Our current logging system works
by filling up a queue that the cl-log lib is then fed from, and sending
lots of data in that queue is way expensive, stop doing that.

Hopefully we don't need to revisit the logs more than that, the other
messages should be few enough not to count much when doing a full load.
This commit is contained in:
Dimitri Fontaine 2014-05-26 16:59:12 +02:00
parent 2637bb7e81
commit e1bf53906d
3 changed files with 9 additions and 3 deletions

View File

@ -20,7 +20,9 @@
(unwind-protect
(loop for i below batch-rows
for copy-string = (aref batch i)
do (log-message :data "> ~s" copy-string)
do (when (or (eq :data *log-min-messages*)
(eq :data *client-min-messages*))
(log-message :data "> ~s" copy-string))
do (cl-postgres:db-write-row copier nil copy-string)
finally (return batch-rows))
(cl-postgres:close-db-writer copier))))

View File

@ -33,7 +33,9 @@
(defun batch-row (row copy queue)
"Add ROW to the reader batch. When the batch is full, provide it to the
writer as the *writer-batch*."
(log-message :data "< ~s" row)
(when (or (eq :data *log-min-messages*)
(eq :data *client-min-messages*))
(log-message :data "< ~s" row))
(let ((oversized? (oversized? *current-batch*)))
(when (or (= (batch-count *current-batch*) *copy-batch-rows*)
oversized?)

View File

@ -332,6 +332,8 @@
(*myconn-port* . ,*myconn-port*)
(*myconn-user* . ,*myconn-user*)
(*myconn-pass* . ,*myconn-pass*)
(*state* . ,*state*))))
(*state* . ,*state*)
(*client-min-messages* . ,*client-min-messages*)
(*log-min-messages* . ,*log-min-messages*))))
"Wrapper around lparallel:make-kernel that sets our usual bindings."
(lp:make-kernel worker-count :bindings bindings))