mxnet 如何查看参数数量


#1

请问mxnet是否支持统计模型参数大小的功能, 如检查NDarray的 element 数量大小,类似torch.numel() 这种


#2

请参考这一部分
https://zh.gluon.ai/chapter_prerequisite/ndarray.html

x = nd.arange(12)
x.size

会得到输出: 12


#3

mxnet 好像也沒有工具能夠直接算 model MAC數量


#4

试试block.summary(sample_input)


#5

试了一下报错了 AttributeError: 'RNNModel' object has no attribute 'summary'

class RNNModel(Block):
    def __init__(self, mode, vocab_size, embed_size, num_hiddens,
                 num_layers, drop_prob=0.5, **kwargs):
        super(RNNModel, self).__init__(**kwargs)
        with self.name_scope():
            self.dropout = nn.Dropout(drop_prob)
            # 将词索引变换成词向量。这些词向量也是模型参数。
            self.embedding = nn.Embedding(
                vocab_size, embed_size, weight_initializer=init.Uniform(0.1))
            if mode == 'rnn_relu':
                self.rnn = rnn.RNN(num_hiddens, num_layers, activation='relu',
                                   dropout=drop_prob, input_size=embed_size)
            elif mode == 'rnn_tanh':
                self.rnn = rnn.RNN(num_hiddens, num_layers, activation='tanh',
                                   dropout=drop_prob, input_size=embed_size)
            elif mode == 'lstm':
                self.rnn = rnn.LSTM(num_hiddens, num_layers,
                                    dropout=drop_prob, input_size=embed_size)
            elif mode == 'gru':
                self.rnn = rnn.GRU(num_hiddens, num_layers, dropout=drop_prob,
                                   input_size=embed_size)
            else:
                raise ValueError("Invalid mode %s. Options are rnn_relu, "
                                 "rnn_tanh, lstm, and gru" % mode)
            self.dense = nn.Dense(vocab_size, in_units=num_hiddens)
            self.num_hiddens = num_hiddens

    def forward(self, inputs, state):
        embedding = self.dropout(self.embedding(inputs))
        output, state = self.rnn(embedding, state)
        output = self.dropout(output)
        output = self.dense(output.reshape((-1, self.num_hiddens)))
        return output, state

    def begin_state(self, *args, **kwargs):
        return self.rnn.begin_state(*args, **kwargs)
    
vocab_size = 10000
embed_size = 280
num_hiddens = 960
num_layers = 3
lr = 1
model = RNNModel('lstm', vocab_size, embed_size, num_hiddens, num_layers,)
model.initialize(init.Xavier())
trainer = gluon.Trainer(model.collect_params(), 'sgd',
                        {'learning_rate': lr, 'momentum': 0, 'wd': 0})
loss = gloss.SoftmaxCrossEntropyLoss() 

model.summary()


#6

需要更新mxnet,可以先用nightly build试试


#7

另外summary里要给一个input,可以是shape跟一般forward一样的随机tensor


#8

model 參數量用大神提供的方法可以成功計算、想問一下有可以直接評估MAC數量的API function嗎?


#9

谢谢,普通的网络可以,但如果有weight dropout的就不行

RuntimeError: Parameter 'rnn0_sequential0_lstm0_l0_h2h_weight' has not been initialized. Note that you should initialize parameters and create Trainer with Block.collect_params() instead of Block.params because the later does not include Parameters of nested child Blocks

#10

MAC只能算理论值,而且MAC少不代表一定就快。比如mobilenet v2 MAC比v1少,但是反而更慢


#11

MAC 是什么?


#12

multiply and accumulate
100M MACs 表示運算完model會做100M乘加


#13

你說的沒錯
但是理論值還是評估的依據
我想MAC數量還是會跟計算時間正相關
@szha 不知道是否有能夠計算的function