r - How to efficiently merge these data.tables -

-

i want create data.table able check missing data. missing data in case not mean there na, entire row left out. need able see of time dependent column values missing level column. important if there lot of missing values or if spread across dataset.

so have 6.000.000x5 data.table (call tablea) containing time dependent variable, id level , value n add final table.

i have table (tableb) 207x2. couples id's factor columns in tablec.

tablec 1.500.000x207 of each of 207 columns correspond id according tableb , rows correspond time dependent variable in tablea.

these tables large , although acquired ram (totalling 8gb) computer keeps swapping away tablec , each write has called back, , gets swapped away again after. swapping consuming time. 1.6 seconds per row of tablea , tablea has 6.000.000 rows operation take more 100 days running non stop..

currently using for-loop loop on rows of tablea. doing no operation for-loop loops instantly. made one-line command looking correct column , row number tablec in tablea , tableb , writing value tablea tablec. broke one-liner system.time analysis , each step takes 0 seconds except writing big tablec. showed writing value table time consuming , looking @ memory use can see huge chunk appearing whenever write happens , disappears finished.

tablea <- data.table("id"=round(runif(200, 1, 100)), "timecounter"=round(runif(200, 1, 50)), "n"=round(rnorm(200, 1, 0.5))) tableb  <- data.table("id"=c(1:100),"realid"=c(100:1)) tsm <- matrix(0,ncol=nrow(tableb), nrow=50) tablec <- as.data.table(tsm) rm(tsm) (row in 1:nrow(tablea)) {   tableccol <- tableb[realid==tablea[row,id],id]   tablecrow <- (tablea[row,timecounter])   val <- tablea[row,n]   tablec[tablecrow,tableccol] <- val }

can advise me on how make operation faster, preventing memory swap @ last step in for-loop?

edit: on advice of @arun took time develop dummy data test on. included in code given above. did not include wanted results because dummy data random , routine work. it's speed problem.

not entirely sure results, give shot dplyr/tidyr packages for, seem more memory efficient loops.

install.packages("dplyr") install.packages("tidyr") library(dplyr) library(tidyr)  tablec <- tablec %>% gather(tablec_id, value, 1:207)

this turns tablec 1,500,000x207 long format 310,500,000x2 table 'tablec_id' , 'tablec_value' columns.

tabled <- tablea %>%      left_join(tableb, c("levelid" = "tableb_id")) %>%      left_join(tablec, c("tableb_value" = "tablec_id")

this couple of packages i've been using of late, , seem efficient, data.table package used management of large tables there useful functions there. i'd take @ sqldf allows query data.frames via sql commands.


Comments

Post a Comment

Popular posts from this blog

matlab - "Contour not rendered for non-finite ZData" -

-
i'm trying plot frequency characteristic equation using ezplot, matlab gives following warning, "contour not rendered non-finite zdata". have used command plot frequency equations warning , plot display empty , not change axis range well. can please help. appreciated. here's code i'm using. % transfer matrix case-i, thin rotor clear all; clc; ei = 1626; l = 0.15; m = 0.44108; = 2.178*10^-4; i_p = 2.205*10^-5; itr = 0.24; i_pr = 0.479; syms p n; f = [1 l*1i l^2/(2*ei)*1i l^3/(6*ei); 0 1 l/ei -l^2/(2*ei)*1i; 0 0 1 -l*1i; 0 0 0 l]; p = [ 1 0 0 0; 0 1 0 0; 0 -it*p^2+i_p*n*p 1 0; -m*p^2 0 0 1]; p_r = [1 0 0 0; 0 1 0 0; 0 -itr*p^2+i_pr*n*p 1 0; -m*p^2 0 0 1]; = f*p*f*p*f*p*f; b = p_r*f*p*f*p*f; r = a(1,2)/a(1,4); a12_p = 0; a22_p = a(2,2)-r*a(2,4); a32_p = a(3,2)-r*a(3,4); a42_p = a(4,2)-r*a(4,4); ap(2,2) = a22_p; ap(3,2) = a32_p; ap(4,2) = a42_p; ap(4,4) = 1; c = b*ap; m = [c(3,2) c(3,4); c(4,2) c(4,...
Read more

delphi - Indy UDP Read Contents of Adata -

-
i using indy udp server read string of data. however, not know how work adata parameter. can around converting binary using bytetobin function below , convert hex using bintohex1 . feel stupid , works slow. know how can directly results without 2 conversions? thanks!! here codes: procedure tform1.idudpserver1udpread(athread: tidudplistenerthread; const adata: tidbytes; abinding: tidsockethandle); var buf: tidbytes; buffer: string; data_received: string; begin if bytestostring(adata) <> 'hello' begin buffer := hextostring('00'); setlength(buf, length(buffer)); copytidstring(buffer, buf, 0); abinding.sendto(abinding.peerip, abinding.peerport, buf); data_received := bintohex1(bytetobin(adata[1]) + bytetobin(adata[2]) + bytetobin(adata[3]) + bytetobin(adata[4]) + bytetobin(adata[5]) + bytetobin(adata[6]) + bytetobin(adata[7]) + bytetobin(adata[8]) + bytetobin(adata[9]) + bytetobin(adata[10]) + bytetobin(adata[11...
Read more

javascript - Any ideas when Firefox is likely to implement lengthAdjust and textLength? -

-
any ideas when firefox implement lengthadjust andtextlength?..... working svg application. lengthadjust , textlength working fine in chrome not working in fire fox . <svg viewbox = "0 0 500 300" version = "1.1"> <defs> <path id="s3" d="m 10,90 q 100,15 200,70" /> </defs> <g> <text font-size = "20"> <textpath xlink:href="#s3" textlength="205"> short text</textpath> </text> <use xlink:href="#s3" fill="none" stroke="black" stroke-width="1"/> </g> </svg> this implemented in firefox text elements (not textpath or tspan) in case that's ok though can move attribute text element. <svg viewbox = "0 0 500 300" version = "1.1"> <defs> ...
Read more