Using theano to implement maximum likelihood learning in neural probability language model Python -
i'm trying implement maximum likelihood learning neural probability language model in python code of log-bilinear model: https://github.com/wenjieguan/log-bilinear-language-models/blob/master/lbl.py
i used grad function in theano compute gradient , try using function train update parameters of model, got errors. here code:
def train(self, sentences, alpha = 0.001, batches = 1000): print('start training...') self.alpha = alpha count = 0 rare = self.vocab['<>'] #print rare q = np.zeros(self.dim, np.float32) #print q delta_context = [np.zeros((self.dim, self.dim), np.float32) in range(self.context) ] #print delta_context delta_feature = np.zeros((len(self.vocab), self.dim), np.float32) #print delta_feature sentence in sentences: sentence = self.start_sen + sentence + self.end_sen pos in range(self.context, len(sentence) ): count += 1 q.fill(0) featurew = [] contextmatrix = [] indices = [] i, r in enumerate(sentence[pos - self.context : pos]): if r == '<_>': continue index = self.vocab.get(r, rare) print index indices.append(index) ri = self.featurevectors[index] #print ri ci = self.contextmatrix[i] #print ci featurew.append(ri) contextmatrix.append(ci) #caculating predicted representation target word q += np.dot(ci, ri) #computing energy function energy = np.exp(np.dot(self.featurevectors, q) + self.biases) #print energy #computing conditional distribution probs = energy / np.sum(energy) #print probs w_index = self.vocab.get(sentence[pos], rare) #computing gradient logprobs = t.log(probs[w_index]) print 'gradient start...' delta_context, delta_feature = t.grad(logprobs, [self.contextmatrix, self.featurevectors]) print 'gradient completed!' train = theano.function( inputs = [self.vocab], outputs = [logprobs], updates=((self.featurevectors,self.featurevectors - self.alpha * delta_feature), (self.contextmatrix,self.contextmatrix - self.alpha * delta_context)), name="train" ) print('training finished!')
i have learnt python , neural probability language model, quite difficult me. me, please! thank you!
Comments
Post a Comment