docker - What causes flume with GCS sink to throw a OutOfMemoryException -

-

i using flume write google cloud storage. flume listens on http:9000. took me time make work (add gcs libaries, use credentials file...) seems communicate on network.

i sending small http request tests, , have plenty of ram available: 

curl -x post -d '[{ "headers" : { timestamp=1417444588182, env=dev, tenant=mytenant, type=mytype }, "body" : "some body one"  }]' localhost:9000

i encounter memory exception on first request (then of course, stops working):

2014-11-28 16:59:47,748 (hdfs-hdfs_sink-call-runner-0) [info - com.google.cloud.hadoop.util.logutil.info(logutil.java:142)] ghfs version: 1.3.0-hadoop2 2014-11-28 16:59:50,014 (sinkrunner-pollingrunner-defaultsinkprocessor) [error - org.apache.flume.sink.hdfs.hdfseventsink.process(hdfseventsink.java:467)] process failed java.lang.outofmemoryerror: java heap space         @ java.io.bufferedoutputstream.<init>(bufferedoutputstream.java:76)         @ com.google.cloud.hadoop.fs.gcs.googlehadoopoutputstream.<init>(googlehadoopoutputstream.java:79)         @ com.google.cloud.hadoop.fs.gcs.googlehadoopfilesystembase.create(googlehadoopfilesystembase.java:820)         @ org.apache.hadoop.fs.filesystem.create(filesystem.java:906)

(see complete stack trace gist full details)

the strange part folders , files created way want, files empty.

gs://my_bucket/dev/mytenant/mytype/2014-12-01/14-36-28.1417445234193.json.tmp

is wrong way configured flume + gcs or bug in gcs.jar ?

where should check gather more data ?

ps : running flume-ng inside docker.

my flume.conf file:

# name components on agent a1.sources = http a1.sinks = hdfs_sink a1.channels = mem  # describe/configure source a1.sources.http.type =  org.apache.flume.source.http.httpsource a1.sources.http.port = 9000  # describe sink a1.sinks.hdfs_sink.type = hdfs a1.sinks.hdfs_sink.hdfs.path = gs://my_bucket/%{env}/%{tenant}/%{type}/%y-%m-%d a1.sinks.hdfs_sink.hdfs.fileprefix = %h-%m-%s a1.sinks.hdfs_sink.hdfs.filesuffix = .json a1.sinks.hdfs_sink.hdfs.round = true a1.sinks.hdfs_sink.hdfs.roundvalue = 10 a1.sinks.hdfs_sink.hdfs.roundunit = minute  # use channel buffers events in memory a1.channels.mem.type = memory a1.channels.mem.capacity = 10000 a1.channels.mem.transactioncapacity = 1000  # bind source , sink channel a1.sources.http.channels = mem a1.sinks.hdfs_sink.channel = mem

related question in flume/gcs journey: what minimal setup needed write hdfs/gs on google cloud storage flume?

when uploading files, gcs hadoop filesystem implementation sets aside large (64mb) write buffer per fsdataoutputstream (file open write). can changed setting "fs.gs.io.buffersize.write" smaller value, in bytes, in core-site.xml. imagine 1mb suffice low-volume log collection.

in addition, check maximum heap size set when launching jvm flume. flume-ng script sets default java_opts value of -xmx20m limit heap 20mb. can set larger value in flume-env.sh (see conf/flume-env.sh.template in flume tarball distribution details).


Comments

Post a Comment

Popular posts from this blog

android - Hide only the Action bar on Scroll not action bar tabs -

-
Image
i getting issue @ trying hide action bar while scrolling down.then while scrolling up,the action bar have shown again. for eg: i referred tutorial .here can see output , respective codes related hide , show action bar. i showing output till now. after scrolling down: the screenshot above shown, hide action bar tab also.but have hide action bar.that's major issue. after scrolling up: the screenshot above shows displays action bar tabs. topratedfragment.java: import info.androidhive.tabsswipe.r; import android.app.actionbar; import android.content.res.typedarray; import android.os.bundle; import android.support.v4.app.fragment; import android.view.layoutinflater; import android.view.view; import android.view.viewgroup; import android.view.viewtreeobserver; import android.view.window; import android.widget.scrollview; public class topratedfragment extends fragment implements viewtreeobserver.onscrollchangedlistener { private float mactionbarheight; ...
Read more

matlab - "Contour not rendered for non-finite ZData" -

-
i'm trying plot frequency characteristic equation using ezplot, matlab gives following warning, "contour not rendered non-finite zdata". have used command plot frequency equations warning , plot display empty , not change axis range well. can please help. appreciated. here's code i'm using. % transfer matrix case-i, thin rotor clear all; clc; ei = 1626; l = 0.15; m = 0.44108; = 2.178*10^-4; i_p = 2.205*10^-5; itr = 0.24; i_pr = 0.479; syms p n; f = [1 l*1i l^2/(2*ei)*1i l^3/(6*ei); 0 1 l/ei -l^2/(2*ei)*1i; 0 0 1 -l*1i; 0 0 0 l]; p = [ 1 0 0 0; 0 1 0 0; 0 -it*p^2+i_p*n*p 1 0; -m*p^2 0 0 1]; p_r = [1 0 0 0; 0 1 0 0; 0 -itr*p^2+i_pr*n*p 1 0; -m*p^2 0 0 1]; = f*p*f*p*f*p*f; b = p_r*f*p*f*p*f; r = a(1,2)/a(1,4); a12_p = 0; a22_p = a(2,2)-r*a(2,4); a32_p = a(3,2)-r*a(3,4); a42_p = a(4,2)-r*a(4,4); ap(2,2) = a22_p; ap(3,2) = a32_p; ap(4,2) = a42_p; ap(4,4) = 1; c = b*ap; m = [c(3,2) c(3,4); c(4,2) c(4,...
Read more

delphi - Indy UDP Read Contents of Adata -

-
i using indy udp server read string of data. however, not know how work adata parameter. can around converting binary using bytetobin function below , convert hex using bintohex1 . feel stupid , works slow. know how can directly results without 2 conversions? thanks!! here codes: procedure tform1.idudpserver1udpread(athread: tidudplistenerthread; const adata: tidbytes; abinding: tidsockethandle); var buf: tidbytes; buffer: string; data_received: string; begin if bytestostring(adata) <> 'hello' begin buffer := hextostring('00'); setlength(buf, length(buffer)); copytidstring(buffer, buf, 0); abinding.sendto(abinding.peerip, abinding.peerport, buf); data_received := bintohex1(bytetobin(adata[1]) + bytetobin(adata[2]) + bytetobin(adata[3]) + bytetobin(adata[4]) + bytetobin(adata[5]) + bytetobin(adata[6]) + bytetobin(adata[7]) + bytetobin(adata[8]) + bytetobin(adata[9]) + bytetobin(adata[10]) + bytetobin(adata[11...
Read more