camel split big sql result in smaller chunks -

-

because of memory limitation need split result sql-component (list<map<column,value>>) smaller chunks (some thousand).

i know 

from(sql:...).split(body()).streaming().to(...)

and know 

.split().tokenize("\n", 1000).streaming()

but latter not working list<map<>> , returning string. there out of box way create chunks? or need add custom aggregator behind split? or there way?

edit additional info requested soilworker: @ moment sql endpoint configured way:

sqlendpoint endpoint = context.getendpoint("sql:select * " + lookuptablename + "?datasource=" + look_up_ds,                                            sqlendpoint.class); // returns complete result in 1 list instead of 1 exchange per line. endpoint.getconsumerproperties().put("useiterator", false); // poll interval endpoint.getconsumerproperties().put("delay", lookup_poll_interval);

the route using should poll once day (we add cronscheduledroutepolicy soon) , fetch complete table (view). data converted csv custom processor , sent via custom component proprietary software. table has 5 columns (small strings) , around 20m entries. don't know if there memory issue. know on local machine 3gb isn't enough. there way approximate memory footprint know if amount of ram enough?

thanks in advance

maxmessagesperpoll result in batches


Comments

Post a Comment

Popular posts from this blog

matlab - "Contour not rendered for non-finite ZData" -

-
i'm trying plot frequency characteristic equation using ezplot, matlab gives following warning, "contour not rendered non-finite zdata". have used command plot frequency equations warning , plot display empty , not change axis range well. can please help. appreciated. here's code i'm using. % transfer matrix case-i, thin rotor clear all; clc; ei = 1626; l = 0.15; m = 0.44108; = 2.178*10^-4; i_p = 2.205*10^-5; itr = 0.24; i_pr = 0.479; syms p n; f = [1 l*1i l^2/(2*ei)*1i l^3/(6*ei); 0 1 l/ei -l^2/(2*ei)*1i; 0 0 1 -l*1i; 0 0 0 l]; p = [ 1 0 0 0; 0 1 0 0; 0 -it*p^2+i_p*n*p 1 0; -m*p^2 0 0 1]; p_r = [1 0 0 0; 0 1 0 0; 0 -itr*p^2+i_pr*n*p 1 0; -m*p^2 0 0 1]; = f*p*f*p*f*p*f; b = p_r*f*p*f*p*f; r = a(1,2)/a(1,4); a12_p = 0; a22_p = a(2,2)-r*a(2,4); a32_p = a(3,2)-r*a(3,4); a42_p = a(4,2)-r*a(4,4); ap(2,2) = a22_p; ap(3,2) = a32_p; ap(4,2) = a42_p; ap(4,4) = 1; c = b*ap; m = [c(3,2) c(3,4); c(4,2) c(4,...
Read more

delphi - Indy UDP Read Contents of Adata -

-
i using indy udp server read string of data. however, not know how work adata parameter. can around converting binary using bytetobin function below , convert hex using bintohex1 . feel stupid , works slow. know how can directly results without 2 conversions? thanks!! here codes: procedure tform1.idudpserver1udpread(athread: tidudplistenerthread; const adata: tidbytes; abinding: tidsockethandle); var buf: tidbytes; buffer: string; data_received: string; begin if bytestostring(adata) <> 'hello' begin buffer := hextostring('00'); setlength(buf, length(buffer)); copytidstring(buffer, buf, 0); abinding.sendto(abinding.peerip, abinding.peerport, buf); data_received := bintohex1(bytetobin(adata[1]) + bytetobin(adata[2]) + bytetobin(adata[3]) + bytetobin(adata[4]) + bytetobin(adata[5]) + bytetobin(adata[6]) + bytetobin(adata[7]) + bytetobin(adata[8]) + bytetobin(adata[9]) + bytetobin(adata[10]) + bytetobin(adata[11...
Read more

qt - How to embed QML toolbar and menubar into QMainWindow -

-
i using qwt library plot data. seems not possible embed qwidget qml quick 2 . so, decided create qmainwindow main window , create toolbar , menubar using quick controls . how should embed qml toolbar , menubar qmainwindow ? you should create qml applicationwindow qml menubar , toolbar main.qml applicationwindow { visible: false menubar: menubar { menu { title: "edit" menuitem { text: "cut" } } } toolbar: toolbar { row { anchors.fill: parent toolbutton { iconsource: "1.png" } } } } main.cpp qapplication app(argc, argv); qqmlapplicationengine engine; engine.load(qurl(qstringliteral("qrc:///main.qml"))); then pointer applicationwindow qwindow *qmlwindow = qobject_cast<qwindow*>(engine.rootobjects().at(0)); create window container, using qwidget::createwindowco...
Read more