r - Max efficiency in removing duplicated rows in data frame -

-

i have large data frame: more 6 million rows, 28 variables of type (num, factors, characters). need remove duplicated rows. however, way identify actual duplicates run check on large character variable (approx 1,000 2,000 characters in each observation). use standard duplicated() function not sure time efficient solution.

is there function or package allows efficiently job ? thank in advance suggestions.

structure(list(city = c("new york", "new york", "new york", "brussels",  "london", "arlington"), prodcategory = structure(c(1l, 1l, 1l,  1l, 1l, 1l), .label = "4", class = "factor"), date = structure(c(16351,  16352, 16351, 16353, 16354, 16355), class = "date"), userid = c("abcd",  "xyzz", "abcd", "abcd", "sdfg", "wedgd"), review = c("in opinion 1 of best pastrami or corned beef sandwiches places in ny (an more). way each sandwich feed whole family days... establishment situated close theatre district , time square. delight see turkey sandwich arrive. wow massive , delicious. ..the celebrity photos awesome ..highly recommend place true taste treat",  "this not usual half-red-lobster place. full experience of super top quality sea food amazingly convenient price basic sandwiches fine cuisine each plate joy.",  "in opinion 1 of best pastrami or corned beef sandwiches places in ny (an more). way each sandwich feed whole family days... establishment situated close theatre district , time square. delight see turkey sandwich arrive. wow massive , delicious. ..the celebrity photos awesome ..highly recommend place true taste treat",  "each time go brussels stop typical brasserie located in historical heart of brussels downtown @ walking distance every interesting place. food great , menu rich , diversified service sharp , fast , pricing reasonable. dont miss typical chocolate cake. should write dont miss... included rich list of belgian beers",  "that call great uk pub food --simple tasty not fat/heavy/greasy (... ok not healthy though) presented service efficient , overall atmosphere deserves stop",  "are fan of house of cards ? have not missed amazing bbq place frank underwood loves go. looks rocklands right you. different atmosphere same kind of yummy meat" )), .names = c("city", "prodcategory", "date", "userid", "review" ), row.names = c(na, -6l), class = "data.frame")

try

library(data.table) setkey(setdt(df), review) res <- unique(df) dim(res) #[1] 5 5

Comments

Post a Comment

Popular posts from this blog

javascript - Any ideas when Firefox is likely to implement lengthAdjust and textLength? -

-
any ideas when firefox implement lengthadjust andtextlength?..... working svg application. lengthadjust , textlength working fine in chrome not working in fire fox . <svg viewbox = "0 0 500 300" version = "1.1"> <defs> <path id="s3" d="m 10,90 q 100,15 200,70" /> </defs> <g> <text font-size = "20"> <textpath xlink:href="#s3" textlength="205"> short text</textpath> </text> <use xlink:href="#s3" fill="none" stroke="black" stroke-width="1"/> </g> </svg> this implemented in firefox text elements (not textpath or tspan) in case that's ok though can move attribute text element. <svg viewbox = "0 0 500 300" version = "1.1"> <defs>
Read more

matlab - "Contour not rendered for non-finite ZData" -

-
i'm trying plot frequency characteristic equation using ezplot, matlab gives following warning, "contour not rendered non-finite zdata". have used command plot frequency equations warning , plot display empty , not change axis range well. can please help. appreciated. here's code i'm using. % transfer matrix case-i, thin rotor clear all; clc; ei = 1626; l = 0.15; m = 0.44108; = 2.178*10^-4; i_p = 2.205*10^-5; itr = 0.24; i_pr = 0.479; syms p n; f = [1 l*1i l^2/(2*ei)*1i l^3/(6*ei); 0 1 l/ei -l^2/(2*ei)*1i; 0 0 1 -l*1i; 0 0 0 l]; p = [ 1 0 0 0; 0 1 0 0; 0 -it*p^2+i_p*n*p 1 0; -m*p^2 0 0 1]; p_r = [1 0 0 0; 0 1 0 0; 0 -itr*p^2+i_pr*n*p 1 0; -m*p^2 0 0 1]; = f*p*f*p*f*p*f; b = p_r*f*p*f*p*f; r = a(1,2)/a(1,4); a12_p = 0; a22_p = a(2,2)-r*a(2,4); a32_p = a(3,2)-r*a(3,4); a42_p = a(4,2)-r*a(4,4); ap(2,2) = a22_p; ap(3,2) = a32_p; ap(4,2) = a42_p; ap(4,4) = 1; c = b*ap; m = [c(3,2) c(3,4); c(4,2) c(4,
Read more

delphi - Indy UDP Read Contents of Adata -

-
i using indy udp server read string of data. however, not know how work adata parameter. can around converting binary using bytetobin function below , convert hex using bintohex1 . feel stupid , works slow. know how can directly results without 2 conversions? thanks!! here codes: procedure tform1.idudpserver1udpread(athread: tidudplistenerthread; const adata: tidbytes; abinding: tidsockethandle); var buf: tidbytes; buffer: string; data_received: string; begin if bytestostring(adata) <> 'hello' begin buffer := hextostring('00'); setlength(buf, length(buffer)); copytidstring(buffer, buf, 0); abinding.sendto(abinding.peerip, abinding.peerport, buf); data_received := bintohex1(bytetobin(adata[1]) + bytetobin(adata[2]) + bytetobin(adata[3]) + bytetobin(adata[4]) + bytetobin(adata[5]) + bytetobin(adata[6]) + bytetobin(adata[7]) + bytetobin(adata[8]) + bytetobin(adata[9]) + bytetobin(adata[10]) + bytetobin(adata[11
Read more