feed.xml

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Libo&#39;s Blog</title>
    <description>Libo Blog</description>
    <link>http://next.sh/</link>
    <atom:link href="http://next.sh/feed.xml" rel="self" type="application/rss+xml"/>
    <pubDate>Fri, 10 Jun 2016 16:19:02 +0000</pubDate>
    <lastBuildDate>Fri, 10 Jun 2016 16:19:02 +0000</lastBuildDate>
    <generator>Jekyll v2.5.3</generator>
    
      <item>
        <title>需要看的论文们</title>
        <description>&lt;p&gt;一共837页&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-text&quot; data-lang=&quot;text&quot;&gt;页数	论文名
9	A Few Useful Things to Know about Machine Learning
5	ADVANCES IN OPTIMIZING RECURRENT NETWORKS
39	Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
9	An Empirical Exploration of Recurrent Network Architectures
8	Backpropagation, Intuitions
11	Batch Normalization-Accelerating Deep Network Training by Reducing Internal Covariate Shift
9	Beyond Short Snippets-Deep Networks for Video Classification
8	CNN Features off-the-shelf- an Astounding Baseline for Recognition
14	Convolutional Neural Networks- Architectures, Convolution, Pooling Layers
10	DeCAF-A Deep Convolutional Activation Feature
8	Deep Inside Convolutional Networks-Visualising Image Classification Models and Saliency Maps
6	Deep Learning using Linear Support Vector Machines
12	Deep Residual Learning for Image Recognition
8	DeepFace-Closing the Gap to Human-Level Performance in Face Verification
8	Deformable Part Models are Convolutional Neural Networks
11	Delving Deep into Rectifiers
11	Delving Deeper into Convolutional Networks for Learning Video Representations
9	Distributed Representations of Words and Phrases
9	Do Convnets Learn Correspondence
9	Dropout Training as Adaptive Regularization
30	Dropout- A Simple Way to Prevent Neural Networks from Overfitting
11	EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES
44	Efficient BackProp
9	Fast R-CNN
13	Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods
14	Faster R-CNN- Towards Real-Time Object Detection with Region Proposal Networks
12	Going Deeper with Convolutions 
4	Hessian matrix - Wikipedia, the free encyclopedia
14	How transferable are features in deep neural networks
10	Image Classification- Data-driven Approach, k-Nearest Neighbor, train:val:test splits 
9	ImageNet Classification with Deep Convolutional
43	ImageNet Large Scale Visual Recognition Challenge
10	Intriguing properties of neural networks
18	LSTM- A Search Space Odyssey
11	Large Scale Distributed Deep Networks
8	Large-scale Video Classification with Convolutional Neural Networks
16	Learning Spatiotemporal Features with 3D Convolutional Networks
12	Linear classification- Support Vector Machine, Softmax
13	Long-term Recurrent Convolutional Networks for Visual Recognition and Description
9	Maxout Networks
8	Neural Networks Part 1- Setting up the Architecture
10	Neural Networks Part 2- Setting up the Data and the Loss
12	Neural Networks Part 3- Learning and Evaluation
12	On the difficulty of training Recurrent Neural Networks
9	Optimization- Stochastic Gradient Descent
16	OverFeat-Integrated Recognition, Localization and Detection using Convolutional Networks
33	Practical Recommendations for Gradient-Based Training of Deep Arch
25	Random Search for Hyper-Parameter Optimization
21	Rich feature hierarchies for accurate object detection and semantic segmentation
14	STRIVING FOR SIMPLICITY- THE ALL CONVOLUTIONAL NET
14	Selective Search for Object Recognition
22	Show, Attend and Tell- Neural Image Caption Generation with Visual Attention
16	Stochastic Gradient Descent Tricks
2	Transfer Learning
11	Two-Stream Convolutional Networks for Action Recognition in Videos
9	Understanding Deep Image Representations by Inverting Them
5	Understanding and Visualizing Convolutional Neural Networks
8	Understanding the difficulty of training deep feedforward neural networks
13	Unit Tests for Stochastic Optimization
14	Very Deep Convolutional Networks for Large-Scale Image Recognition
11	Visualizing and Understanding Convolutional Networks
6	What I learned from competing against a ConvNet on ImageNet
16	What makes for effective detection proposals
7	video process&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
</description>
        <pubDate>Sun, 03 Apr 2016 23:39:51 +0000</pubDate>
        <link>http://next.sh/blog/2016/04/03/those-thesis/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/04/03/those-thesis/</guid>
        
        <category>Thesis</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>视频处理笔记</title>
        <description>&lt;p&gt;Feature-based approaches to Activity Recognition&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dense trajectories and motion boundary descriptors for action recognition&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Wang et al., 2013&lt;/p&gt;

&lt;p&gt;Action Recognition with Improved Trajectories&lt;/p&gt;

&lt;p&gt;Wang and Schmid, 2013&lt;/p&gt;

&lt;p&gt;用来track视频中什么有变化&lt;/p&gt;

&lt;p&gt;[T. Brox and J. Malik, “Large displacement optical flow: Descriptor matching in variational motion estimation,” 2011]&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-one-idea-of-convnet.png&quot; alt=&quot;flow&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Q: What if the input is now a small chunk of video? E.g. [227x227x3x15] ?&lt;/p&gt;

&lt;p&gt;A: Extend the convolutional filters in time, perform spatio-temporal convolutions!E.g. can have 11x11xT filters, where T = 2..15.&lt;/p&gt;

&lt;p&gt;filter不仅在空间上移动，还在时间上移动&lt;/p&gt;

&lt;h2 id=&quot;spatio-temporal-convnets&quot;&gt;Spatio-Temporal ConvNets&lt;/h2&gt;

&lt;p&gt;Before AlexNets
[3D Convolutional Neural Networks for Human Action Recognition, Ji et al., 2010]&lt;/p&gt;

&lt;p&gt;[Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011]&lt;/p&gt;

&lt;p&gt;After&lt;/p&gt;

&lt;p&gt;[Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014]&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-spatio-temporal-video.png&quot; alt=&quot;spatio-temporal convolutions&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Single Frame -&amp;gt;非常好的一种，比较推荐先尝试这个&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-frame-diff-performance.png&quot; alt=&quot;Model &amp;amp;&amp;amp; Gain&quot; /&gt;&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;C3D architechure&lt;/p&gt;

&lt;p&gt;in time &amp;amp; VGG &amp;amp; work well&lt;/p&gt;

&lt;p&gt;[Learning Spatiotemporal Features with 3D Convolutional Networks, Tran et al. 2015]&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;2D convolution&lt;/p&gt;

&lt;p&gt;Simonyan 是VGG的提出人&lt;/p&gt;

&lt;p&gt;[Two-Stream Convolutional Networks for Action Recognition in Videos, Simonyan and Zisserman 2014]&lt;/p&gt;

&lt;p&gt;Two-stream version works much better than either alone.&lt;/p&gt;

&lt;h3 id=&quot;longer-term-events&quot;&gt;Longer Term Events&lt;/h3&gt;

&lt;p&gt;使用LSTM&lt;/p&gt;

&lt;p&gt;[Long-term Recurrent Convolutional Networks for Visual Recognition and Description, Donahue et al., 2015]&lt;/p&gt;

&lt;p&gt;[Beyond Short Snippets: Deep Networks for Video Classification, Ng et al., 2015]&lt;/p&gt;

&lt;h4 id=&quot;section&quot;&gt;奇葩！！！&lt;/h4&gt;

&lt;p&gt;(This paper was way ahead of its time. Cited 65 times.)&lt;/p&gt;

&lt;p&gt;Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011&lt;/p&gt;

&lt;h2 id=&quot;section-1&quot;&gt;三种构架&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Model temporal motion locally (3D CONV)&lt;/li&gt;
  &lt;li&gt;Model temporal motion globally (LSTM / RNN)&lt;/li&gt;
  &lt;li&gt;Fusions of both approaches at the same&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;idea:快进快退&lt;/p&gt;

&lt;h2 id=&quot;long-time-spatio-temporal-convnets&quot;&gt;Long-time Spatio-Temporal ConvNets&lt;/h2&gt;

&lt;p&gt;[Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016]&lt;/p&gt;

&lt;p&gt;！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！
很重要，这个应该能用&lt;/p&gt;

&lt;p&gt;GRU是一个LSTM的简化算法&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-long-time-spatio-temporal.png&quot; alt=&quot;long time spatio temporal convnets&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Infinite (in theory) temporal extent&lt;/p&gt;

&lt;p&gt;(neurons that are function of all video frames in the past)&lt;/p&gt;

&lt;h3 id=&quot;section-2&quot;&gt;注意&lt;/h3&gt;

&lt;p&gt;在做视频的时候，事先考虑Spatial Temporal Video ConvNet&lt;/p&gt;

&lt;p&gt;确定需要local motion（3D Conv）还是global motion（LSTM）&lt;/p&gt;

&lt;p&gt;Try out using Optical Flow in a second stream&lt;/p&gt;

&lt;p&gt;Try out GRU-RCN! (imo best model)&lt;/p&gt;

&lt;h1 id=&quot;unsupervised&quot;&gt;Unsupervised&lt;/h1&gt;

&lt;p&gt;Clustering, dimensionality reduction, feature learning, generative models, etc&lt;/p&gt;

&lt;p&gt;Autoencoders&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Vanilla		Traditional: feature learning&lt;/li&gt;
  &lt;li&gt;Variational: generate samples&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;● Generative Adversarial Networks: Generate samples&lt;/p&gt;

&lt;h2 id=&quot;encoder&quot;&gt;Encoder&lt;/h2&gt;

&lt;p&gt;输入Data，经过Encoder，然后去找Features&lt;/p&gt;

&lt;p&gt;Feature一般比Data小，所以可以用于降维&lt;/p&gt;

&lt;p&gt;然后通过Decoder，然后通过Features重建Input Data&lt;/p&gt;

&lt;p&gt;Decoder和Encoder 的Weight是共享的&lt;/p&gt;

&lt;hr /&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;After training, throw away decoder!&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Use encoder to initialize a supervised model&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;hr /&gt;

&lt;p&gt;三层NN，不过输入输出一样，因为需要中间的Features，要用到其他地方&lt;/p&gt;

&lt;p&gt;PCA可以解决一部分，更多是Reconstruction&lt;/p&gt;

&lt;h2 id=&quot;variational-autoencoder-encoder&quot;&gt;Variational Autoencoder: Encoder&lt;/h2&gt;

&lt;p&gt;Kingma and Welling, “Auto-Encoding Variational Bayes”, ICLR 2014&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-var-encoder.png&quot; alt=&quot;Encoder&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-var-autoencoder.png&quot; alt=&quot;Variational Autoencoder&quot; /&gt;&lt;/p&gt;

&lt;p&gt;完全没听懂==&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-math.png&quot; alt=&quot;no idea&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014&lt;/p&gt;

&lt;h3 id=&quot;generative-adversarial-nets-multiscale&quot;&gt;Generative Adversarial Nets: Multiscale&lt;/h3&gt;

&lt;p&gt;Generator is an upsampling network with fractionally-strided convolutions&lt;/p&gt;

&lt;p&gt;Discriminator is a convolutional network&lt;/p&gt;

&lt;p&gt;Radford et al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR 2016&lt;/p&gt;

&lt;p&gt;！！！！！！！！！！！！
超牛逼&lt;/p&gt;

&lt;p&gt;Radford et al,ICLR 2016&lt;/p&gt;

&lt;h3 id=&quot;generative-adversarial-nets-vector-math&quot;&gt;Generative Adversarial Nets: Vector Math&lt;/h3&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-generate-smile.png&quot; alt=&quot;smile&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-generate-glass.png&quot; alt=&quot;glass&quot; /&gt;&lt;/p&gt;

&lt;h2 id=&quot;put-everything-together&quot;&gt;Put everything together&lt;/h2&gt;

&lt;p&gt;Dosovitskiy and Brox, “Generating Images with Perceptual Similarity Metrics based on Deep Networks”,arXiv 2016&lt;/p&gt;

</description>
        <pubDate>Sun, 27 Mar 2016 11:01:39 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/27/lecture-14-video-and-unsupervised-learning-stanford-cs231n/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/27/lecture-14-video-and-unsupervised-learning-stanford-cs231n/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>分割、Attention Model与空间变化 笔记</title>
        <description>&lt;p&gt;Szegedy et al, Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, arXiv 2016&lt;/p&gt;

&lt;p&gt;GoogLeNet-v4&lt;/p&gt;

&lt;p&gt;Top 5，error 3.08%&lt;/p&gt;

&lt;hr /&gt;

&lt;h1 id=&quot;segmentation&quot;&gt;Segmentation&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Semantic Segmentation（不知道有几个，只是对每个像素label了）&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Instance Segmentation（SDS，对每个个体都要区分，不同的人也要分）&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;semantic-segmentation&quot;&gt;Semantic Segmentation&lt;/h2&gt;

&lt;p&gt;Figure credit: Shotton et al, “TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context”, IJCV 2007&lt;/p&gt;

&lt;h2 id=&quot;instance-segmentation&quot;&gt;Instance Segmentation&lt;/h2&gt;

&lt;p&gt;Figure credit: Dai et al, “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, arXiv 2015&lt;/p&gt;

&lt;h1 id=&quot;semantic-segmentation-1&quot;&gt;Semantic Segmentation&lt;/h1&gt;

&lt;p&gt;输出图像因为Pooling，会比之前的小&lt;/p&gt;

&lt;h3 id=&quot;image-pyramid&quot;&gt;image pyramid&lt;/h3&gt;

&lt;p&gt;Resize to multiple different sizes&lt;/p&gt;

&lt;p&gt;each scales -&amp;gt; run one cnn per scale -&amp;gt; up scale ouputs and concatenate&lt;/p&gt;

&lt;p&gt;法2&lt;/p&gt;

&lt;p&gt;RGB三个通道&lt;/p&gt;

&lt;p&gt;Apply CNN once&lt;/p&gt;

&lt;p&gt;More iterations improve results&lt;/p&gt;

&lt;p&gt;Pinheiro and Collobert, “Recurrent Convolutional Neural Networks for Scene Labeling”, ICML 2014&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Learnable upsampling!&lt;/strong&gt;&lt;/p&gt;

&lt;h4 id=&quot;skip-connections&quot;&gt;skip connections&lt;/h4&gt;

&lt;p&gt;Better results&lt;/p&gt;

&lt;p&gt;从pool3或者pool4跳到最后&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-sematic-upsampling-deconvolution.png&quot; alt=&quot;upsampling&quot; /&gt;&lt;/p&gt;

&lt;h3 id=&quot;deconvolution&quot;&gt;Deconvolution&lt;/h3&gt;

&lt;p&gt;!!!Input gives weight for filter&lt;/p&gt;

&lt;p&gt;convolution的时候，stride1，deconvolution的时候stride2&lt;/p&gt;

&lt;p&gt;重叠的地方，相加&lt;/p&gt;

&lt;p&gt;Same as backward pass for normal convolution!&lt;/p&gt;

&lt;p&gt;“inverse of convolution”&lt;/p&gt;

&lt;p&gt;名字：&lt;/p&gt;

&lt;p&gt;convolution transpose,backward strided convolution,1/2 strided convolution,upconvolution&lt;/p&gt;

&lt;p&gt;名字的争论的论文。。。。&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Im et al, “Generating images with recurrent adversarial networks”, arXiv 2016&lt;/li&gt;
  &lt;li&gt;Radford et al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;instance-segmentation-1&quot;&gt;Instance Segmentation&lt;/h2&gt;

&lt;p&gt;Input-&amp;gt;Region Proposal (Segment Proposal) -&amp;gt; External Segment proposal&lt;/p&gt;

&lt;p&gt;然后两条路，一条Feature Extraction另一条 R-CNN&lt;/p&gt;

&lt;p&gt;-&amp;gt;Region CLassification&lt;/p&gt;

&lt;p&gt;-&amp;gt;Refinement&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;判断foreground还是background&lt;/p&gt;

&lt;p&gt;Hariharan et al, “Hypercolumns for Object Segmentation and Fine-grained Localization”, CVPR 2015&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;Google Instance Segmentation COCO2015获胜&lt;/p&gt;

&lt;p&gt;Dai et al, “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, arXiv 2015&lt;/p&gt;

&lt;p&gt;Region proposal network (RPN)&lt;/p&gt;

&lt;p&gt;然后使用Rol warping pooling&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-r-cnn-instance-seg-coco2015.png&quot; alt=&quot;faster R-CNN &amp;amp;&amp;amp; other to captioning&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-coco-2015-performance.png&quot; alt=&quot;COCO 2015 performance&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;attention-models&quot;&gt;Attention Models&lt;/h1&gt;

&lt;p&gt;Attention Models&lt;/p&gt;

&lt;p&gt;每次处理并不是全部处理所有的Input，每次处理只处理最Attention的那部分&lt;/p&gt;

&lt;p&gt;最近很火&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;R-CNN&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-rcnn-captioning.png&quot; alt=&quot;R-CNN&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Soft Attention&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-27-soft-attention-for-captioning.png&quot; alt=&quot;Soft Attention&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;reinforcement learning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;但是不知道在自由图片上的效果&lt;/p&gt;

&lt;h2 id=&quot;soft-attention-for-translation&quot;&gt;Soft Attention for Translation&lt;/h2&gt;

&lt;p&gt;Sequence -&amp;gt; Sequence&lt;/p&gt;

&lt;p&gt;Video captioning,attention over input frames:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Yao et al, “Describing Videos by Exploiting Temporal Structure”, ICCV 2015&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Image, question to answer,attention over image:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Xu and Saenko, “Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering”, arXiv 2015&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Zhu et al, “Visual7W: Grounded Question Answering in Images”, arXiv 2015&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;rnn-handwriting&quot;&gt;RNN handwriting&lt;/h3&gt;

&lt;p&gt;Graves, “Generating Sequences with Recurrent Neural Networks”, arXiv 2013&lt;/p&gt;

&lt;p&gt;Demo &lt;a href=&quot;http://www.cs.toronto.edu/~graves/handwriting.html&quot;&gt;http://www.cs.toronto.edu/~graves/handwriting.html&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&quot;spatial-transformer-networks&quot;&gt;Spatial Transformer Networks&lt;/h2&gt;

&lt;p&gt;Jaderberg et al, “Spatial Transformer Networks”, NIPS 2015&lt;/p&gt;

&lt;p&gt;Soft attention:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Easy to implement: produce distribution over input locations, reweight features and feed as input&lt;/li&gt;
  &lt;li&gt;Attend to arbitrary input locations using spatial transformer networks&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Hard attention:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Attend to a single input location&lt;/li&gt;
  &lt;li&gt;Can’t use gradient descent!&lt;/li&gt;
  &lt;li&gt;Need reinforcement learning&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Selectively paying attention to different parts of the image&lt;/strong&gt;&lt;/p&gt;

</description>
        <pubDate>Sat, 26 Mar 2016 16:09:21 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/26/lecture-13-segmentation-and-attention-stanford-cs231n/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/26/lecture-13-segmentation-and-attention-stanford-cs231n/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>Caffe Torch TensorFlow</title>
        <description>&lt;p&gt;Written in C++&lt;/p&gt;

&lt;p&gt;Has Python and MATLAB bindings&lt;/p&gt;

&lt;p&gt;Good for training or finetuning feedforward model&lt;/p&gt;

&lt;h2 id=&quot;caffe&quot;&gt;Caffe&lt;/h2&gt;

&lt;p&gt;不需要写代码有时候==，例如ResNet&lt;/p&gt;

&lt;p&gt;Document可能outdated&lt;/p&gt;

&lt;p&gt;Four Major Classes：&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Blob : Stores data and derivatives(weights,data,labels) [data diffs],[gpu cpu]&lt;/li&gt;
  &lt;li&gt;Layer : input(bottom) blob -&amp;gt; output(top) blob.&lt;/li&gt;
  &lt;li&gt;Net : Many Layers. computes gradients via forward/backward&lt;/li&gt;
  &lt;li&gt;Solver : Uses gradients to update weights&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;protocol-buffers&quot;&gt;Protocol Buffers&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;Typed Json from google&lt;/li&gt;
  &lt;li&gt;Define “message types” in .proto files&lt;/li&gt;
  &lt;li&gt;Serialize instances to text files .prototxt&lt;/li&gt;
  &lt;li&gt;Compile classes for different languages&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;caffe-training--finetuning&quot;&gt;Caffe: Training / Finetuning&lt;/h2&gt;

&lt;p&gt;No need to write code!&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Convert data (run a script)&lt;/li&gt;
  &lt;li&gt;Define net (edit prototxt)&lt;/li&gt;
  &lt;li&gt;Define solver (edit prototxt)&lt;/li&gt;
  &lt;li&gt;Train (with pretrained weights) (run a script)&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;convert-data&quot;&gt;Convert Data&lt;/h3&gt;

&lt;p&gt;LMDB:&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;&lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;path/to/image.jpeg&lt;span class=&quot;o&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;label&lt;span class=&quot;o&quot;&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;DataLayer 直接从LMDB读是最方便的&lt;/p&gt;

&lt;p&gt;用&lt;a href=&quot;https://github.com/BVLC/caffe/blob/master/tools/convert_imageset.cpp&quot;&gt;https://github.com/BVLC/caffe/blob/master/tools/convert_imageset.cpp&lt;/a&gt;来创建LMDB&lt;/p&gt;

&lt;p&gt;使用h5py去创建HDF5&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;ImageDataLayer: Read from image files&lt;/li&gt;
  &lt;li&gt;WindowDataLayer: For detection&lt;/li&gt;
  &lt;li&gt;HDF5Layer: Read from HDF5 file&lt;/li&gt;
  &lt;li&gt;From memory, using Python interface&lt;/li&gt;
  &lt;li&gt;All of these are harder to use (except Python)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;define-net&quot;&gt;Define Net&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;.prototxt can get ugly for big models&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;ResNet-152 prototxt is 6775 lines long!&lt;/p&gt;

&lt;p&gt;一般可以来写Python来生成&lt;/p&gt;

&lt;hr /&gt;

&lt;h4 id=&quot;prototxt&quot;&gt;直接修改 prototxt&lt;/h4&gt;

&lt;p&gt;Same name: weights copied&lt;/p&gt;

&lt;p&gt;Different name: weights reinitialized&lt;/p&gt;

&lt;h3 id=&quot;define-solver&quot;&gt;Define Solver&lt;/h3&gt;

&lt;p&gt;If finetuning, copy existing solver. prototxt file&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Change net to be your net&lt;/li&gt;
  &lt;li&gt;Change snapshot_prefix to your output&lt;/li&gt;
  &lt;li&gt;Reduce base learning rate (divide by 100)&lt;/li&gt;
  &lt;li&gt;Maybe change max_iter and snapshot&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;例如learning rate，可以改成几个值，然后去用&lt;/p&gt;

&lt;h3 id=&quot;train&quot;&gt;Train&lt;/h3&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;./build/tools/caffe train &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  -gpu &lt;span class=&quot;m&quot;&gt;0&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  -model path/to/trainval.prototxt &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  -solver path/to/solver.prototxt &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  -weights path/to/pretrained_weights.caffemodel&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;-gpu -1 是CPU mode&lt;/p&gt;

&lt;p&gt;-gpu all 是多GPU 并行&lt;/p&gt;

&lt;h2 id=&quot;model-zoo&quot;&gt;Model Zoo&lt;/h2&gt;

&lt;p&gt;AlexNet, VGG, GoogLeNet, ResNet, plus others&lt;/p&gt;

&lt;h2 id=&quot;caffe-python-interface&quot;&gt;Caffe: Python Interface&lt;/h2&gt;

&lt;p&gt;Not much documentation…&lt;/p&gt;

&lt;p&gt;Read the code! Two most important files:&lt;/p&gt;

&lt;p&gt;● caffe/python/caffe/_caffe.cpp:&lt;/p&gt;

&lt;p&gt;○ Exports Blob, Layer, Net, and Solver classes&lt;/p&gt;

&lt;p&gt;● caffe/python/caffe/pycaffe.py&lt;/p&gt;

&lt;p&gt;○ Adds extra methods to Net class&lt;/p&gt;

&lt;p&gt;Good for:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Interfacing with numpy&lt;/li&gt;
  &lt;li&gt;Extract features: Run net forward&lt;/li&gt;
  &lt;li&gt;Compute gradients: Run net backward (DeepDream, etc)&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Define layers&lt;/strong&gt; in Python with numpy (CPU only)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;如果定义layer的话，只能是CPU的&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;注意下⬆️这里&lt;/strong&gt;&lt;/p&gt;

&lt;h2 id=&quot;caffe-pros--cons&quot;&gt;Caffe Pros / Cons&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;(+) Good for feedforward networks&lt;/li&gt;
  &lt;li&gt;(+) Good for finetuning existing networks&lt;/li&gt;
  &lt;li&gt;(+) Train models without writing any code!&lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+) Python interface is pretty useful!&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;(-) Need to write C++ / CUDA for new GPU layers（在Caffe上写新的Layer很麻烦）&lt;/li&gt;
  &lt;li&gt;(-) Not good for recurrent networks
= (-) Cumbersome for big networks (GoogLeNet, ResNet)&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;torch&quot;&gt;Torch&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;From NYU + IDIAP&lt;/li&gt;
  &lt;li&gt;Written in C and Lua&lt;/li&gt;
  &lt;li&gt;Used a lot a Facebook, DeepMind&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;在GPU上运行超简单&lt;/p&gt;

&lt;h3 id=&quot;lua&quot;&gt;必须用Lua来写&lt;/h3&gt;

&lt;p&gt;&lt;a href=&quot;http://tylerneylon.com/a/learn-lua/&quot;&gt;15分钟学会Lua&lt;/a&gt;&lt;/p&gt;

&lt;h3 id=&quot;tensors-like-numpy-arrays&quot;&gt;Tensors like numpy arrays&lt;/h3&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-lua&quot; data-lang=&quot;lua&quot;&gt;&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;cutorch&amp;#39;&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;cunn&amp;#39;&lt;/span&gt;

&lt;span class=&quot;kd&quot;&gt;local&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;torch.CudaTensor&amp;#39;&lt;/span&gt;

&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/torch/torch7/blob/master/doc/tensor.md&quot;&gt;https://github.com/torch/torch7/blob/master/doc/tensor.md Document&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&quot;torch--nn&quot;&gt;Torch :: nn&lt;/h2&gt;

&lt;p&gt;nn module lets you easily build and train neural nets&lt;/p&gt;

&lt;p&gt;Build a two-layer ReLU net&lt;/p&gt;

&lt;h2 id=&quot;torch--cunn&quot;&gt;Torch : cunn&lt;/h2&gt;

&lt;p&gt;Import a few new packages&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-lua&quot; data-lang=&quot;lua&quot;&gt;&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;torch&amp;#39;&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;cutorch&amp;#39;&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;nn&amp;#39;&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;cunn&amp;#39;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Cast network and criterion&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-lua&quot; data-lang=&quot;lua&quot;&gt;&lt;span class=&quot;n&quot;&gt;net&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;crit&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Cast data and labels&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-lua&quot; data-lang=&quot;lua&quot;&gt;&lt;span class=&quot;kd&quot;&gt;local&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;D&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;kd&quot;&gt;local&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;y&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;C&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id=&quot;torch-optim&quot;&gt;Torch: optim&lt;/h2&gt;

&lt;p&gt;update rules: momentum, Adam, etc&lt;/p&gt;

&lt;p&gt;Import optim package&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;requre &lt;span class=&quot;s1&quot;&gt;&amp;#39;optim&amp;#39;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Write a callback function that returns loss and gradients&lt;/p&gt;

&lt;p&gt;state variable holds hyperparameters, cached values, etc; pass it to adams&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-lua&quot; data-lang=&quot;lua&quot;&gt;&lt;span class=&quot;n&quot;&gt;optim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;adm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;f&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;weights&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;state&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id=&quot;torch-modules&quot;&gt;Torch: Modules&lt;/h2&gt;

&lt;p&gt;Caffe has Nets and Layers;&lt;/p&gt;

&lt;p&gt;Torch just has Modules&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Forward / backward written in Lua using Tensor methods&lt;/li&gt;
  &lt;li&gt;Modules are classes written in Lua; easy to read and write&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Same code runs on CPU / GPU&lt;/p&gt;

&lt;h2 id=&quot;container&quot;&gt;Container&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Sequential&lt;/li&gt;
  &lt;li&gt;ConcatTable&lt;/li&gt;
  &lt;li&gt;ParallelTable&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-torch-container.png&quot; alt=&quot;Container&quot; /&gt;&lt;/p&gt;

&lt;h2 id=&quot;torch-nngraph&quot;&gt;Torch: nngraph&lt;/h2&gt;

&lt;p&gt;Use nngraph to build modules that combine their inputs in complex ways&lt;/p&gt;

&lt;h2 id=&quot;torch-package-management&quot;&gt;Torch: Package Management&lt;/h2&gt;

&lt;p&gt;loadcaffe: Load pretrained Caffe models: AlexNet, VGG, some others&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/szagoruyko/loadcaffe&quot;&gt;https://github.com/szagoruyko/loadcaffe&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;GoogLeNet v1: &lt;a href=&quot;https://github.com/soumith/inception.torch&quot;&gt;https://github.com/soumith/inception.torch&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;GoogLeNet v3: &lt;a href=&quot;https://github.com/Moodstocks/inception-v3.torch&quot;&gt;https://github.com/Moodstocks/inception-v3.torch&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;ResNet: &lt;a href=&quot;https://github.com/facebook/fb.resnet.torch&quot;&gt;https://github.com/facebook/fb.resnet.torch&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&quot;torch-package-management-1&quot;&gt;Torch: Package Management&lt;/h2&gt;

&lt;p&gt;like pip : luarocks&lt;/p&gt;

&lt;h2 id=&quot;torch-other-useful-packages&quot;&gt;Torch: Other useful packages&lt;/h2&gt;

&lt;p&gt;● torch.cudnn: Bindings for NVIDIA cuDNN kernels&lt;/p&gt;

&lt;p&gt;● torch-hdf5: Read and write HDF5 files from Torch&lt;/p&gt;

&lt;p&gt;● lua-cjson: Read and write JSON files from Lua,&lt;a href=&quot;https://luarocks.org/modules/luarocks/lua-cjson&quot;&gt;https://luarocks.org/modules/luarocks/lua-cjson&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;● cltorch, clnn: OpenCL backend for Torch, and port of nn&lt;/p&gt;

&lt;p&gt;● torch-autograd: Automatic differentiation; sort of like more powerful nngraph, similar to Theano or TensorFlow&lt;/p&gt;

&lt;p&gt;https://github.com/twitter/torch-autograd&lt;/p&gt;

&lt;p&gt;● fbcunn: Facebook: FFT conv, multi-GPU (DataParallel, ModelParallel)&lt;a href=&quot;https://github.com/facebook/fbcunn&quot;&gt;https://github.com/facebook/fbcunn&lt;/a&gt;&lt;/p&gt;

&lt;h3 id=&quot;torch-typical-workflow&quot;&gt;Torch: Typical Workflow&lt;/h3&gt;

&lt;p&gt;Step 1: Preprocess data; usually use a Python script to dump data to HDF5&lt;/p&gt;

&lt;p&gt;Step 2: Train a model in Lua / Torch; read from HDF5 datafile, save trained model to disk&lt;/p&gt;

&lt;p&gt;Step 3: Use trained model for something, often with an evaluation script&lt;/p&gt;

&lt;p&gt;https://github.com/jcjohnson/torch-rnn&lt;/p&gt;

&lt;p&gt;Step 1: Preprocess data; usually use a Python script to dump data to HDF5 (https://github.com/jcjohnson/torch-rnn/blob/master/scripts/preprocess.py)&lt;/p&gt;

&lt;p&gt;Step 2: Train a model in Lua / Torch; read from HDF5 datafile, save trained model to disk (https://github.com/jcjohnson/torch-rnn/blob/master/train.lua)&lt;/p&gt;

&lt;p&gt;Step 3: Use trained model for something, often with an evaluation script (https://github.com/jcjohnson/torch-rnn/blob/master/sample.lua)&lt;/p&gt;

&lt;h1 id=&quot;torch-pros--cons&quot;&gt;Torch: Pros / Cons&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;(+) Lots of modular pieces that are easy to combine&lt;/li&gt;
  &lt;li&gt;(+) Easy to write your own layer types and run on GPU&lt;/li&gt;
  &lt;li&gt;(+) Most of the library code is in Lua, easy to read&lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+) Lots of pretrained models!&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;(-) Not great for RNNs&lt;/li&gt;
  &lt;li&gt;(-) Less plug-and-play than Caffe. You usually write your own training code&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;theano&quot;&gt;Theano&lt;/h2&gt;

&lt;p&gt;Python&lt;/p&gt;

&lt;p&gt;http://deeplearning.net/software/theano/&lt;/p&gt;

&lt;p&gt;Embracing computation graphs, symbolic computation&lt;/p&gt;

&lt;p&gt;High-level wrappers: Keras, Lasagne&lt;/p&gt;

&lt;h3 id=&quot;compile-a-function&quot;&gt;Compile a function&lt;/h3&gt;

&lt;p&gt;在运行的时候，编译新的算法&lt;/p&gt;

&lt;p&gt;produces c from x, y, z (generates code)&lt;/p&gt;

&lt;h3 id=&quot;symbolically&quot;&gt;symbolically！！！！！！！！&lt;/h3&gt;

&lt;p&gt;Theano computes gradients for us symbolically!&lt;/p&gt;

&lt;h2 id=&quot;theano-computing-gradients&quot;&gt;Theano: Computing Gradients&lt;/h2&gt;

&lt;p&gt;Problem: Shipping weights and gradients to CPU on every iteration to update…&lt;/p&gt;

&lt;p&gt;解决方案：Theano: Shared Variables&lt;/p&gt;

&lt;p&gt;Define weights as shared variables that persist in the graph between calls; initialize with numpy arrays&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;n&quot;&gt;w1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;theano&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;shared&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;1e-3&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;D&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;H&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&amp;#39;w1&amp;#39;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Function includes an update that updates weights on every call&lt;/p&gt;

&lt;p&gt;To train the net, just call function repeatedly!!!!!!!!!!!!!!&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-Theano-Train.png&quot; alt=&quot;repeat function&quot; /&gt;&lt;/p&gt;

&lt;h2 id=&quot;theano-1&quot;&gt;还有很多Theano的特性&lt;/h2&gt;

&lt;h3 id=&quot;lasagne-high-level-wrapper&quot;&gt;Lasagne: High Level Wrapper&lt;/h3&gt;

&lt;p&gt;writes the update rule for you&lt;/p&gt;

&lt;h3 id=&quot;keras-high-level-wrapper&quot;&gt;Keras: High level wrapper&lt;/h3&gt;

&lt;p&gt;makes common things easy to do&lt;/p&gt;

&lt;p&gt;(Also supports TensorFlow backend)&lt;/p&gt;

&lt;h2 id=&quot;theano-pros--cons&quot;&gt;Theano: Pros / Cons&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;(+) Python + numpy&lt;/li&gt;
  &lt;li&gt;(+) Computational graph is nice abstraction&lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+) RNNs fit nicely in computational graph&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+) High level wrappers (Keras, Lasagne) ease the pain&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;(-) Raw Theano is somewhat low-level&lt;/li&gt;
  &lt;li&gt;(-) Error messages can be unhelpful&lt;/li&gt;
  &lt;li&gt;(-) Large models can have long compile times&lt;/li&gt;
  &lt;li&gt;(-) Much “fatter” than Torch; more magic&lt;/li&gt;
  &lt;li&gt;(-) Patchy support for pretrained models&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;编译超级慢&lt;/p&gt;

&lt;p&gt;超级复杂&lt;/p&gt;

&lt;p&gt;错误超级乱&lt;/p&gt;

&lt;h2 id=&quot;tensorflow&quot;&gt;TensorFlow&lt;/h2&gt;

&lt;p&gt;Very similar to Theano - all about computation graphs&lt;/p&gt;

&lt;p&gt;Easy visualizations (TensorBoard)&lt;/p&gt;

&lt;p&gt;Multi-GPU and multi-node training&lt;/p&gt;

&lt;h3 id=&quot;tensorflow-tensorboard&quot;&gt;TensorFlow: Tensorboard&lt;/h3&gt;

&lt;p&gt;可视化&lt;/p&gt;

&lt;p&gt;卧槽！！！&lt;/p&gt;

&lt;p&gt;可以做直接画图&lt;/p&gt;

&lt;p&gt;而且可以直接画出整个流程图&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-tensorflow-graph1.png&quot; alt=&quot;TensorFlow board graph1&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-tensorflow-graph2.png&quot; alt=&quot;TensorFlow board graph2&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;tensorflow-pros--cons&quot;&gt;TensorFlow: Pros / Cons&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;(+) Python + numpy&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+) Computational graph abstraction, like Theano; great for RNNs&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;(+) Much faster compile times than Theano&lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+) Slightly more convenient than raw Theano?&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+) TensorBoard for visualization&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;(+) Data AND model parallelism; best of all frameworks&lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;(+/-) Distributed models, but not open-source yet&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;(-) Slower than other frameworks right now&lt;/li&gt;
  &lt;li&gt;(-) Much “fatter” than Torch; more magic&lt;/li&gt;
  &lt;li&gt;(-) Not many pretrained models&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;caffe-vs-torch-vs-theano-vs-tensorflow&quot;&gt;Caffe vs Torch vs Theano vs TensorFlow&lt;/h1&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-caffe-vs-torch-vs-theano-vs-tensorflow.png&quot; alt=&quot;caffe vs torch vs theano vs tensorflow&quot; /&gt;&lt;/p&gt;

&lt;hr /&gt;

&lt;h4 id=&quot;segmentation-classify-every-pixel&quot;&gt;Segmentation? (Classify every pixel)&lt;/h4&gt;

&lt;ul&gt;
  &lt;li&gt;Need pretrained model (Caffe, Torch, Lasagna)&lt;/li&gt;
  &lt;li&gt;Need funny loss function&lt;/li&gt;
  &lt;li&gt;If loss function exists in Caffe: Use Caffe&lt;/li&gt;
  &lt;li&gt;If you want to write your own loss: Use Torch&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;object-detection&quot;&gt;Object Detection?&lt;/h4&gt;

&lt;p&gt;-&amp;gt; Use Caffe + Python or Torch&lt;/p&gt;

&lt;h4 id=&quot;language-modeling-with-new-rnn-structure&quot;&gt;Language modeling with new RNN structure?&lt;/h4&gt;

&lt;p&gt;-&amp;gt; Use Theano or TensorFlow&lt;/p&gt;

&lt;h4 id=&quot;implement-batchnorm&quot;&gt;Implement BatchNorm?&lt;/h4&gt;

&lt;p&gt;-&amp;gt; Implement efficient backward pass? Use Torch&lt;/p&gt;

&lt;h1 id=&quot;recommendation&quot;&gt;Recommendation&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Feature extraction / finetuning existing models: Use Caffe&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Complex uses of pretrained models: Use Lasagne or Torch&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Write your own layers: Use Torch&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Crazy RNNs: Use Theano or Tensorflow&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Huge model, need model parallelism: Use TensorFlow&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;neon™&lt;/p&gt;

&lt;p&gt;The Fastest Deep Learning Framework&lt;/p&gt;

&lt;p&gt;neon™ is open source and can be deployed on CPUs, GPUs or custom Nervana hardware. It supports all the commonly used models including convnets, MLPs, RNNs, LSTMs and autoencoder&lt;/p&gt;

</description>
        <pubDate>Sat, 26 Mar 2016 12:17:16 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/26/lecture-12-cnn-caffe-torch-tensorflow-stanford-cs231n/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/26/lecture-12-cnn-caffe-torch-tensorflow-stanford-cs231n/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        <category>Caffe</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>CNN使用的一些细节</title>
        <description>&lt;p&gt;Data augmentation: artificially expand your data&lt;/p&gt;

&lt;p&gt;Transfer learning: CNNs without huge data&lt;/p&gt;

&lt;h2 id=&quot;cnncaffe&quot;&gt;CNN实战和Caffe&lt;/h2&gt;

&lt;h3 id=&quot;data-augmentation&quot;&gt;Data Augmentation&lt;/h3&gt;

&lt;p&gt;如何Transform Image&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;改变像素不改变Label&lt;/li&gt;
  &lt;li&gt;Train on transformed data&lt;/li&gt;
  &lt;li&gt;VERY widely used&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;常见：&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Horizontal flips&lt;/li&gt;
  &lt;li&gt;Random crops/scales&lt;/li&gt;
  &lt;li&gt;Color jitter（eg:Randomly jitter contrast）&lt;/li&gt;
&lt;/ol&gt;

&lt;ul&gt;
  &lt;li&gt;translation&lt;/li&gt;
  &lt;li&gt;rotation&lt;/li&gt;
  &lt;li&gt;stretching&lt;/li&gt;
  &lt;li&gt;shearing,&lt;/li&gt;
  &lt;li&gt;lens distortions&lt;/li&gt;
  &lt;li&gt;其他go crazy&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;random-crops&quot;&gt;Random crops&lt;/h4&gt;

&lt;p&gt;Training: sample random crops / scales&lt;/p&gt;

&lt;p&gt;ResNet
1. Pick random L in range [256, 480]
2. Resize training image, short side = L
3. Sample random 224 x 224 patch&lt;/p&gt;

&lt;p&gt;Testing: average a fixed set of crops&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Resize image at 5 scales: {224, 256, 384, 480, 640}&lt;/li&gt;
  &lt;li&gt;For each size, use 10 224 x 224 crops: 4 corners + center, + flips&lt;/li&gt;
&lt;/ol&gt;

&lt;h4 id=&quot;color-jitter&quot;&gt;Color jitter&lt;/h4&gt;

&lt;p&gt;(As seen in [Krizhevsky et al. 2012], ResNet, etc)&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Apply PCA to all [R, G, B] pixels in training set&lt;/li&gt;
  &lt;li&gt;Sample a “color offset” along principal component directions&lt;/li&gt;
  &lt;li&gt;Add offset to all pixels of a training image&lt;/li&gt;
&lt;/ol&gt;

&lt;h2 id=&quot;section&quot;&gt;常用&lt;/h2&gt;

&lt;ol&gt;
  &lt;li&gt;Training: Add random noise&lt;/li&gt;
  &lt;li&gt;Testing: Marginalize over the noise&lt;/li&gt;
&lt;/ol&gt;

&lt;h2 id=&quot;you-need-a-lot-of-a-data-if-you-want-to-trainuse-cnns&quot;&gt;“You need a lot of a data if you want to train/use CNNs”&lt;/h2&gt;

&lt;p&gt;并不是==，&lt;strong&gt;Transfer Learning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;下载已经训练好的，然后从后面开始替换，重新训练&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;因为前面的ConvNet，更倾向于一些底层特征&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Train on Imagenet&lt;/li&gt;
  &lt;li&gt;small dataset feature extractor（去掉FC+softmax，只训练这两层）&lt;/li&gt;
  &lt;li&gt;Medium dataset: Fineturning（从后面再去掉几层ConvNet）&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tip: use only ~1/10th of the original learning rate in finetuning top layer, and ~1/100th on intermediate layers&lt;/p&gt;

&lt;h4 id=&quot;thesis&quot;&gt;Thesis&lt;/h4&gt;

&lt;p&gt;CNN Features off-the-shelf: an Astounding Baseline for Recognition [Razavian et al, 2014]&lt;/p&gt;

&lt;p&gt;DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition[Donahue&lt;em&gt;, Jia&lt;/em&gt;, et al.,2013]&lt;/p&gt;

&lt;h3 id=&quot;transfer-learning-&quot;&gt;Transfer Learning 规则&lt;/h3&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-transfer-learning.png&quot; alt=&quot;Transfer Learning&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Model Zoo&lt;/p&gt;

&lt;h2 id=&quot;all-about-convolutions&quot;&gt;All About Convolutions&lt;/h2&gt;

&lt;h3 id=&quot;how-to-stack-them&quot;&gt;How to stack them&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Suppose we stack two 3x3 conv layers (stride 1)&lt;/li&gt;
  &lt;li&gt;Each neuron sees 3x3 region of previous activation map&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;small-fitlers&quot;&gt;small fitlers&lt;/h4&gt;

&lt;p&gt;Suppose input is H x W x C and we use convolutions with C filters&lt;/p&gt;

&lt;p&gt;7x7&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;有多少 Weights：$ = C \times (7 \times 7 \times C) = 49 C^2$&lt;/li&gt;
  &lt;li&gt;多少次计算(乘加)：= (H x W x C) x (7 x 7 x C) = 49 $H W C^2$&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;3x3&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;有多少 Weights：$ = 3 \times C \times (3 \times 3 \times C) = 27 C^2$&lt;/li&gt;
  &lt;li&gt;多少次计算(乘加)：= 3 x (H x W x C) x (3 x 3 x C) 27 $H W C^2$&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;相对于7x7，小的3x3：计算量少，非线性多，相对好&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why stop at 3 x 3 filters? Why not try 1 x 1?&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;“bottleneck” 1 x 1 conv to reduce dimension&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;使用Conv 1x1，C/2 filters&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;3 x 3 conv at reduced dimension&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;使用Conv 3x3，C/2 filters&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Restore dimension with another 1 x 1 conv&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;使用 Conv 1x1, C filters&lt;/p&gt;

&lt;p&gt;[Seen in Lin et al, “Network in Network”,GoogLeNet, ResNet]&lt;/p&gt;

&lt;p&gt;1x1，只需要 $3.25 C^2$
3x3，需要$9 C^2&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-why-stop.png&quot; alt=&quot;Why stop at 3x3&quot; /&gt;&lt;/p&gt;

&lt;p&gt;或者在1x1中，不使用3x3，使用1x3,3x1&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-why-still-use-3x3-in-1x1.png&quot; alt=&quot;why-still-use-3x3-in-1x1&quot; /&gt;&lt;/p&gt;

&lt;h4 id=&quot;googlenet&quot;&gt;GoogLeNet&lt;/h4&gt;

&lt;p&gt;Szegedy et al, “Rethinking the Inception Architecture for Computer Vision”&lt;/p&gt;

&lt;p&gt;大量使用了1x1、3x3&lt;/p&gt;

&lt;h4 id=&quot;recap&quot;&gt;Recap&lt;/h4&gt;

&lt;ul&gt;
  &lt;li&gt;Replace large convolutions (5 x 5, 7 x 7) with stacks of 3 x 3 convolutions&lt;/li&gt;
  &lt;li&gt;1 x 1 “bottleneck” convolutions are very efficient&lt;/li&gt;
  &lt;li&gt;Can factor N x N convolutions into 1 x N and N x 1&lt;/li&gt;
  &lt;li&gt;All of the above give &lt;strong&gt;fewer parameters, less compute, more nonlinearity&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;how-to-compute-conv&quot;&gt;How to compute conv&lt;/h3&gt;

&lt;h4 id=&quot;im2col&quot;&gt;im2col&lt;/h4&gt;

&lt;p&gt;将convolution转为matrix multiplication&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-im2col.png&quot; alt=&quot;im2col&quot; /&gt;&lt;/p&gt;

&lt;h4 id=&quot;fft&quot;&gt;FFT&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Not much speedup for 3x3 filters&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Convolution Theorem: The convolution of f and g is equal&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;F(f*g) = F(f) \cdot F(g)&lt;/script&gt;

&lt;p&gt;Using the Fast Fourier Transform, we can compute the Discrete Fourier transform of an N-dimensional vector in O(N log N) time (also extends to 2D images)&lt;/p&gt;

&lt;p&gt;使用FFT来计算Conv&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Compute FFT of weights: F(W)&lt;/li&gt;
  &lt;li&gt;Compute FFT of image: F(X)&lt;/li&gt;
  &lt;li&gt;Compute elementwise product: F(W) ○ F(X)&lt;/li&gt;
  &lt;li&gt;Compute inverse FFT: Y = F-1(F(W) ○ F(X))&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Vasilache et al, Fast Convolutional Nets With fbfft: A GPU Performance Evaluation&lt;/p&gt;

&lt;h4 id=&quot;fast-algorithms&quot;&gt;Fast Algorithms&lt;/h4&gt;

&lt;p&gt;还没有特别流行，但是很有意思&lt;/p&gt;

&lt;p&gt;Lavin and Gray, “Fast Algorithms for Convolutional Neural Networks”, 2015&lt;/p&gt;

&lt;h2 id=&quot;implementation-detail&quot;&gt;Implementation Detail&lt;/h2&gt;

&lt;p&gt;NVIDIA vs AMD&lt;/p&gt;

&lt;p&gt;NVIDIA is much more common for deep learning&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Introduced new Titan X GPU by bragging about AlexNet benchmarks&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;CUDA (NVIDIA only)
○ Write C code that runs directly on the GPU
○ Higher-level APIs: cuBLAS, cuFFT, cuDNN, etc&lt;/p&gt;

&lt;p&gt;网课 Udacity: Intro to Parallel Programming &lt;a href=&quot;https://www.udacity.com/course/cs344&quot;&gt;https://www.udacity.com/course/cs344&lt;/a&gt;
○ For deep learning just use existing libraries&lt;/p&gt;

&lt;h3 id=&quot;gpu&quot;&gt;即便用GPU，训练也是很慢&lt;/h3&gt;

&lt;p&gt;VGG: ~2-3 weeks training with 4 GPUs&lt;/p&gt;

&lt;p&gt;ResNet 101: 2-3 weeks with 4 GPUs&lt;/p&gt;

&lt;h3 id=&quot;parallel&quot;&gt;Parallel&lt;/h3&gt;

&lt;p&gt;Alex Krizhevsky, “One weird trick for parallelizing convolutional neural networks”&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-26-conv-parallel.png&quot; alt=&quot;conv parallel promising&quot; /&gt;&lt;/p&gt;

&lt;p&gt;[Large Scale Distributed Deep Networks, Jeff Dean et al., 2013]&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Google: Synchronous vs Async&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Abadi et al, “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems”&lt;/p&gt;

&lt;p&gt;TensorFlow 更多是Distribution&lt;/p&gt;

&lt;h1 id=&quot;bottlenecks&quot;&gt;Bottlenecks&lt;/h1&gt;

&lt;p&gt;1、 GPU - CPU communication is a bottleneck.（尤其是小批量的）&lt;/p&gt;

&lt;p&gt;解决方法&lt;/p&gt;

&lt;p&gt;CPU data prefetch+augment thread running while GPU performs forward/backward pass&lt;/p&gt;

&lt;p&gt;Caffe已经做了&lt;/p&gt;

&lt;p&gt;2、CPU - disk bottleneck&lt;/p&gt;

&lt;p&gt;Hard disk is slow to read from&lt;/p&gt;

&lt;p&gt;解决方法，使用SSD&lt;/p&gt;

&lt;p&gt;3、GPU memory bottleneck&lt;/p&gt;

&lt;p&gt;Titan X: 12 GB &amp;lt;- currently the max GTX 980 Ti: 6 GB&lt;/p&gt;

&lt;p&gt;eg：AlexNet: ~3GB needed with batch size 256&lt;/p&gt;

&lt;p&gt;4、Floating Point Precision&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;64 bit “double” precision is default in a lot of programming&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;strong&gt;32 bit “single” precision is typically used for CNNs for performance&lt;/strong&gt;&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Prediction: 16 bit “half” precision &lt;strong&gt;will be the new standard&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Already supported in cuDNN&lt;/li&gt;
  &lt;li&gt;Nervana（公司名） fp16 kernels are the fastest right now&lt;/li&gt;
  &lt;li&gt;Hardware support in next-gen NVIDIA cards (Pascal)&lt;/li&gt;
  &lt;li&gt;Not yet supported in torch\caffe&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;How low can we go?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Gupta et al, 2015：Train with 16-bit fixed point with stochastic rounding&lt;/p&gt;

&lt;p&gt;Gupta et al, “Deep Learning with Limited Numerical Precision”, ICML 2015&lt;/p&gt;

&lt;p&gt;Courbariaux et al, “Training Deep Neural Networks with Low Precision Multiplications”, ICLR 2015&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Train with 10-bit activations, 12-bit parameter updates&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;！！！！2016年的！！！！！！！！&lt;/p&gt;

&lt;p&gt;Courbariaux et al, “BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1”, arXiv 2016&lt;/p&gt;

&lt;p&gt;Courbariaux and Bengio, February 9 2016:&lt;/p&gt;

&lt;p&gt;● Train with 1-bit activations and weights!
● All activations and weights are +1 or -1
● Fast multiplication with bitwise XNOR
● (Gradients use higher precision)&lt;/p&gt;

</description>
        <pubDate>Sat, 26 Mar 2016 10:37:34 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/26/lecture-11-convnets-in-practice-stanford-cs231n/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/26/lecture-11-convnets-in-practice-stanford-cs231n/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>RNN、Image Captioning和LSTM笔记 Stanford CS231n</title>
        <description>&lt;h2 id=&quot;recurrent-neural-network&quot;&gt;Recurrent Neural Network&lt;/h2&gt;

&lt;p&gt;CNN&lt;/p&gt;

&lt;p&gt;fixed size image -&amp;gt; fixed size of vector label&lt;/p&gt;

&lt;p&gt;-one to one&lt;/p&gt;

&lt;p&gt;Image Captioning&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;image -&amp;gt; sequence of words&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Sentiment Classification&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;sequence of words -&amp;gt; sentiment(positive or negative)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Machine Translation&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;seq -&amp;gt; seq of words&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Video classification on frame level&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;before the frame&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Process sequentially&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;fixed inputs vs fixed outputs&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Visual Attention Ba et al&lt;/li&gt;
  &lt;li&gt;Draw a RNN for Image Generation Gregor et al&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;rnn&quot;&gt;RNN&lt;/h3&gt;

&lt;p&gt;It has state.through time,feed input vector into RNN.State changing as received input vectors.&lt;/p&gt;

&lt;p&gt;usually we can predict a vector at some time steps.&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;y 
^
&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;
RNN&amp;lt;-self 
^
&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;
x&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;base on RNN to predict&lt;/p&gt;

&lt;p&gt;has states&lt;/p&gt;

&lt;p&gt;We can process a sequence of vectors X by applying a recurrence formula at every time step.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;h_t = f_W (h_{t-1} , x_t)&lt;/script&gt;

&lt;p&gt;$h_t$ is new state,$h_{t-1}$ is old state,$f_W$ is &lt;strong&gt;the recurrent function&lt;/strong&gt; with parameters $W$.&lt;/p&gt;

&lt;p&gt;训练出$f_W$里的$W$，使用的时候，use $f_W$ at every single step no matter how long the input or output sequences are.&lt;/p&gt;

&lt;h3 id=&quot;vanilla-rnn&quot;&gt;Vanilla RNN&lt;/h3&gt;

&lt;p&gt;Simplest.&lt;/p&gt;

&lt;p&gt;a single hidden vector $h$.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;h_t = tanh(W_{hh} h_{t-1} + W_{xh} x_t) \\
y_t = W_{hy} h_t&lt;/script&gt;

&lt;p&gt;feed character one at a time into RNN.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-RNN-intuition.png&quot; alt=&quot;Character-level language model example&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;numpy&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;np&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# data I/O&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;open&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&amp;#39;input.txt&amp;#39;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&amp;#39;r&amp;#39;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;read&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# should be simple plain text file&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;chars&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;set&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;data_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;chars&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;print&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&amp;#39;data has &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;%d&lt;/span&gt;&lt;span class=&quot;s&quot;&gt; characters, &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;%d&lt;/span&gt;&lt;span class=&quot;s&quot;&gt; unique.&amp;#39;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;char_to_ix&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;i&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;enumerate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;chars&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;ix_to_char&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;enumerate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;chars&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# hyperparameters&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;100&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# size of hidden layer of neurons&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;seq_length&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;25&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# number of steps to unroll the RNN for 必须在内存里都存了，用chunk&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;learning_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;1e-1&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# model parameters&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;Wxh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.01&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# input to hidden&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;Whh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.01&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# hidden to hidden&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;Why&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.01&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# hidden to output&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;bh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# hidden bias&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;by&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# output bias&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;lossFun&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;targets&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hprev&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
  &lt;span class=&quot;sd&quot;&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class=&quot;sd&quot;&gt;  inputs,targets are both list of integers.&lt;/span&gt;
&lt;span class=&quot;sd&quot;&gt;  hprev is Hx1 array of initial hidden state&lt;/span&gt;
&lt;span class=&quot;sd&quot;&gt;  returns the loss, gradients on model parameters, and last hidden state&lt;/span&gt;
&lt;span class=&quot;sd&quot;&gt;  &amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;xs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ys&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ps&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{},&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{},&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{},&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{}&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;copy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hprev&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;
  &lt;span class=&quot;c&quot;&gt;# forward pass (compute loss)&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;t&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;xrange&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;xs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# encode in 1-of-k representation&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;xs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;][&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tanh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Wxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;xs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Whh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;bh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# hidden state (RNN的部分)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;ys&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Why&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;by&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# unnormalized log probabilities for next chars&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;ps&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;exp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ys&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;exp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ys&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]))&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# probabilities for next chars&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;log&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ps&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;][&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;targets&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# softmax (cross-entropy loss)&lt;/span&gt;
  &lt;span class=&quot;c&quot;&gt;# backward pass: compute gradients going backwards&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;dWxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhy&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Wxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Whh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Why&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;dbh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dby&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;by&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;dhnext&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;t&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;reversed&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;xrange&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dy&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;copy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ps&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;targets&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# backprop into y&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dWhy&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dby&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dy&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Why&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dhnext&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# backprop into h&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dhraw&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dh&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# backprop through tanh nonlinearity&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dbh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dhraw&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dWxh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dhraw&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;xs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dWhh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dhraw&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;dhnext&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Whh&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dhraw&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dparam&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dWxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dbh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dby&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;clip&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dparam&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;out&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dparam&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# clip to mitigate exploding gradients&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dbh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dby&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
  &lt;span class=&quot;c&quot;&gt;# weight maxtrices&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;sample&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;seed_ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt; 
  &lt;span class=&quot;sd&quot;&gt;&amp;quot;&amp;quot;&amp;quot; &lt;/span&gt;
&lt;span class=&quot;sd&quot;&gt;  sample a sequence of integers from the model &lt;/span&gt;
&lt;span class=&quot;sd&quot;&gt;  h is memory state, seed_ix is seed letter for first time step&lt;/span&gt;
&lt;span class=&quot;sd&quot;&gt;  &amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;seed_ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;ixes&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[]&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;t&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;xrange&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tanh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Wxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Whh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;bh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;y&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Why&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;by&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;exp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;exp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;ix&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;choice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ravel&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;())&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;ixes&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;append&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ixes&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;mWxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mWhh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mWhy&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Wxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Whh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Why&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;mbh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mby&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;by&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# memory variables for Adagrad&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;smooth_loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;log&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;1.0&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;/&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;seq_length&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# loss at iteration 0&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# Main Loop，a batch(25) of data&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;c&quot;&gt;# prepare inputs (we&amp;#39;re sweeping from left to right in steps seq_length long)&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;seq_length&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;or&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;==&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; 
    &lt;span class=&quot;n&quot;&gt;hprev&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zeros&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# reset RNN memory&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# go from start of data&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;char_to_ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;seq_length&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]]&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;targets&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;char_to_ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ch&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;seq_length&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]]&lt;/span&gt;

  &lt;span class=&quot;c&quot;&gt;# sample from the model now and then&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;100&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;==&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;sample_ix&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sample&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hprev&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;200&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;txt&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&amp;#39;&amp;#39;&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;join&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ix_to_char&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ix&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sample_ix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;print&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&amp;#39;----&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt; &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;%s&lt;/span&gt;&lt;span class=&quot;s&quot;&gt; &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;----&amp;#39;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;txt&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

  &lt;span class=&quot;c&quot;&gt;# forward seq_length characters through the net and fetch gradient&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dbh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dby&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hprev&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;lossFun&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;targets&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hprev&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;smooth_loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;smooth_loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;0.999&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;0.001&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;100&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;==&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;print&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&amp;#39;iter &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;%d&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;, loss: &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;%f&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&amp;#39;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;smooth_loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# print progress&lt;/span&gt;
  
  &lt;span class=&quot;c&quot;&gt;# perform parameter update with Adagrad&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;param&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dparam&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mem&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;zip&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Wxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Whh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Why&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;bh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;by&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; 
                                &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dWxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dWhy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dbh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dby&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; 
                                &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mWxh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mWhh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mWhy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mbh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mby&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;mem&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dparam&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dparam&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;param&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;learning_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dparam&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sqrt&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mem&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;1e-8&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# adagrad update&lt;/span&gt;

  &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;seq_length&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# move data pointer&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# iteration counter&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id=&quot;searching-for-interpretable-cells&quot;&gt;Searching for interpretable cells&lt;/h2&gt;

&lt;p&gt;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot;&gt;karpathy 的博客&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;[Visualizing and Understanding Recurrent Networks, Andrej Karpathy&lt;em&gt;, Justin Johnson&lt;/em&gt;, Li Fei-Fei]&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;quote detection cell&lt;/li&gt;
  &lt;li&gt;line length tracking cell&lt;/li&gt;
  &lt;li&gt;if statement cell&lt;/li&gt;
  &lt;li&gt;quote/comment cell&lt;/li&gt;
  &lt;li&gt;code depth cell&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Image Captioning Sentence Datasets&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;mscoco.org&quot;&gt;Microsoft COCO Tsung-Yi Lin et al. 2014&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&quot;lstm&quot;&gt;LSTM&lt;/h2&gt;

&lt;p&gt;Show Attend and Tell, Xu et al., 2015&lt;/p&gt;

&lt;p&gt;得到RNN的结果，用这个结果去看下一步去哪里找&lt;/p&gt;

&lt;p&gt;RNN attends spatially to different parts of images while generating each word of the sentence&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-RNN-to-LSTM.png&quot; alt=&quot;RNN with attention over the image&quot; /&gt;&lt;/p&gt;

&lt;p&gt;RNN stack layers,&lt;strong&gt;depth&lt;/strong&gt; and &lt;strong&gt;time&lt;/strong&gt;&lt;/p&gt;

&lt;h3 id=&quot;lstm-long-short-term-memory&quot;&gt;LSTM (Long short term memory)&lt;/h3&gt;

&lt;p&gt;[Hochreiter et al., 1997]&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-LSTM-formula.png&quot; alt=&quot;LSTM formula&quot; /&gt;&lt;/p&gt;

&lt;p&gt;$c_t$ cell state vector&lt;/p&gt;

&lt;h4 id=&quot;rnn-vs-lstm&quot;&gt;RNN vs LSTM&lt;/h4&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-RNN-vs-LSTM.png&quot; alt=&quot;RNN vs LSTM&quot; /&gt;&lt;/p&gt;

&lt;p&gt;ignoring forget gates&lt;/p&gt;

&lt;p&gt;就和“PlainNets” vs. ResNets一样，一个是add，一个是直接传递&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-ResNet-vs-PlainNet.png&quot; alt=&quot;ResNet vs plainNet&quot; /&gt;&lt;/p&gt;

&lt;p&gt;RNN 和 LSTM 视觉区别，gradient的变化，RNN很多gradient在传递过来承重逐渐的die了。LSTM is super highway pipeline.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://imgur.com/gallery/vaNahKE&quot;&gt;视觉区别视频&lt;/a&gt;&lt;/p&gt;

&lt;h3 id=&quot;rnn-&quot;&gt;RNN 的不稳定性&lt;/h3&gt;

&lt;p&gt;因为代码里有个&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;&lt;span class=&quot;nv&quot;&gt;Whh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; np.random.randn&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;H,H&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;
...
dss&lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;t&lt;span class=&quot;o&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;hs&lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;t&lt;span class=&quot;o&quot;&gt;]&lt;/span&gt; &amp;gt; 0&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt; * dhs&lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;t&lt;span class=&quot;o&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
dhs&lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;t-1&lt;span class=&quot;o&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; np.dot&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;Whh.T,dss&lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;t&lt;span class=&quot;o&quot;&gt;])&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;导致
if the largest eigenvalue is &amp;gt; 1, gradient will explode （如果太大了，控制住==，大于多少就不增加了）&lt;/p&gt;

&lt;p&gt;if the largest eigenvalue is &amp;lt; 1, gradient will vanish&lt;/p&gt;

&lt;h3 id=&quot;thesis&quot;&gt;Thesis&lt;/h3&gt;

&lt;p&gt;[On the difficulty of training Recurrent Neural Networks, Pascanu et al., 2013]&lt;/p&gt;

&lt;p&gt;[LSTM: A Search Space Odyssey,Greff et al., 2015]&lt;/p&gt;

&lt;p&gt;[An Empirical Exploration of Recurrent Network Architectures,Jozefowicz et al., 2015]&lt;/p&gt;

&lt;h4 id=&quot;gru-changed-lstm-&quot;&gt;GRU changed LSTM 变量少了&lt;/h4&gt;

&lt;p&gt;GRU [Learning phrase representations using rnn encoder-decoder for statistical machine translation, Cho et al. 2014]&lt;/p&gt;

&lt;p&gt;Raw RNN 工作并不是特别好&lt;/p&gt;

</description>
        <pubDate>Thu, 24 Mar 2016 18:31:09 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/24/lecture-10-recurrent-neural-networks-and-image-captioning-lstm-stanford-cs231n/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/24/lecture-10-recurrent-neural-networks-and-image-captioning-lstm-stanford-cs231n/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>RNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>ConvNet Image定位、监测、可视化笔记CS231n</title>
        <description>&lt;h2 id=&quot;computer-vision-task&quot;&gt;Computer Vision task&lt;/h2&gt;

&lt;p&gt;Localization和Detection的一大区别是，Localization是定位一个，而Detection是多个&lt;/p&gt;

&lt;p&gt;Localization:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Find a fixed number of objects (one or many)&lt;/li&gt;
  &lt;li&gt;L2 regression from CNN features to box coordinates&lt;/li&gt;
  &lt;li&gt;Much simpler than detection; consider it for your projects!&lt;/li&gt;
  &lt;li&gt;Overfeat: Regression + efficient sliding window with FC -&amp;gt; conv conversion&lt;/li&gt;
  &lt;li&gt;Deeper networks do better&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Object Detection:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Find a variable number of objects by classifying image regions&lt;/li&gt;
  &lt;li&gt;Before CNNs: dense multiscale sliding window (HoG, DPM)&lt;/li&gt;
  &lt;li&gt;Avoid dense sliding window with region proposals&lt;/li&gt;
  &lt;li&gt;R-CNN: Selective Search + CNN classification / regression&lt;/li&gt;
  &lt;li&gt;Fast R-CNN: Swap order of convolutions and region extraction&lt;/li&gt;
  &lt;li&gt;Faster R-CNN: Compute region proposals within the network&lt;/li&gt;
  &lt;li&gt;Deeper networks do better&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;idea-1-localization-as-regression&quot;&gt;Idea #1: Localization as Regression&lt;/h3&gt;

&lt;p&gt;use box coordinates (4 numbers)&lt;/p&gt;

&lt;p&gt;Step 1: Train (or download) a classification model&lt;/p&gt;

&lt;p&gt;Image -&amp;gt; Conv &amp;amp; Pooling -&amp;gt; FInal conv feature map -&amp;gt; FC -&amp;gt; class scores -&amp;gt; softmax&lt;/p&gt;

&lt;p&gt;Step 2: Attach new fully-connected “regression head” to the network&lt;/p&gt;

&lt;p&gt;“Regression head”&lt;/p&gt;

&lt;p&gt;Image -&amp;gt; Conv &amp;amp; Pooling -&amp;gt; FInal conv feature map -&amp;gt; FC -&amp;gt; Box coordinates（real number）&lt;/p&gt;

&lt;p&gt;Step 3: Train the regression head only with SGD and L2 loss&lt;/p&gt;

&lt;p&gt;Step 4: At test time use both heads&lt;/p&gt;

&lt;p&gt;如果有C个Classes&lt;/p&gt;

&lt;p&gt;Classification head: C numbers&lt;/p&gt;

&lt;p&gt;Regression head: C x 4 numbers&lt;/p&gt;

&lt;h3 id=&quot;where-to-attach-the-regression-head&quot;&gt;Where to attach the regression head?&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;After conv layers: Overfeat, VGG&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;After last FC layer: DeepPose, R-CNN&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;aside-localizing-multiple-objects&quot;&gt;Aside: Localizing multiple objects&lt;/h4&gt;

&lt;p&gt;K x 4 numbers
(one box per object)&lt;/p&gt;

&lt;p&gt;Human Pose Estimation&lt;/p&gt;

&lt;p&gt;找到关节 joints&lt;/p&gt;

&lt;p&gt;Regress (x, y) for each joint from last fully-connected layer of AlexNet&lt;/p&gt;

&lt;p&gt;Toshev and Szegedy, “DeepPose: Human Pose Estimation via Deep Neural Networks”, CVPR 2014&lt;/p&gt;

&lt;h3 id=&quot;idea-2-sliding-window&quot;&gt;Idea #2: Sliding Window&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Run classification + regression network at multiple locations on a high-resolution image&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Convert fully-connected layers into convolutional layers for efficient computation&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Combine classifier and
regressor predictions across all scales for final prediction&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;L2 distance？？？？？？？？？？？？？？&lt;/p&gt;

&lt;h3 id=&quot;overfeat&quot;&gt;Overfeat&lt;/h3&gt;

&lt;p&gt;Classification Head &amp;amp; regression head&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sermanet et al, “Integrated Recognition, Localization and Detection using Convolutional Networks”, ICLR 2014&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Greedily merge boxes and scores (details in paper)&lt;/p&gt;

&lt;p&gt;示意图：&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24Overfeat.png&quot; alt=&quot;Overfeat&quot; /&gt;&lt;/p&gt;

&lt;h3 id=&quot;efficient-sliding-window-overfeat&quot;&gt;Efficient Sliding Window: Overfeat&lt;/h3&gt;

&lt;p&gt;Efficient sliding window by converting fully- connected layers into convolutions&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-Efficient-Overfeat.png&quot; alt=&quot;Efficient Overfeat&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Softmax loss&lt;/p&gt;

&lt;p&gt;Euclidean loss&lt;/p&gt;

&lt;h4 id=&quot;section&quot;&gt;现状&lt;/h4&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;AlexNet: Localization method not published&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Overfeat: Multiscale convolutional regression with box merging&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;VGG: Same as Overfeat(只是deeper了), but fewer scales and locations; simpler method, gains all due to deeper features&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;ResNet: Different localization method (RPN) and much deeper features&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;regression--classification-are-two-hammers&quot;&gt;Regression 和 Classification are two hammers&lt;/h2&gt;

&lt;h2 id=&quot;detection&quot;&gt;Detection&lt;/h2&gt;

&lt;p&gt;用Classification&lt;/p&gt;

&lt;p&gt;如何定window size&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Literally try them all&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Q:Need to test many positions and scales&lt;/p&gt;

&lt;p&gt;A: if your classifier is a fast enough,just do it.&lt;/p&gt;

&lt;p&gt;用Linear classifier 然后使用HOG特征，因为快，所以随便大小&lt;/p&gt;

&lt;p&gt;Dalal and Triggs, “Histograms of Oriented Gradients for Human Detection”, CVPR 2005 Slide credit: Ross Girshick&lt;/p&gt;

&lt;h2 id=&quot;deformable-parts-model-dpm&quot;&gt;Deformable Parts Model (DPM)&lt;/h2&gt;

&lt;p&gt;Felzenszwalb et al, “Object Detection with Discriminatively Trained Part Based Models”, PAMI 2010&lt;/p&gt;

&lt;p&gt;也比较快&lt;/p&gt;

&lt;p&gt;Girschick et al,“Deformable Part Models are Convolutional Neural Networks”, CVPR 2015&lt;/p&gt;

&lt;p&gt;–
Problem: Need to test many positions and scales, and use a computationally demanding classifier (CNN)&lt;/p&gt;

&lt;p&gt;Solution: Only look at a tiny subset of possible positions&lt;/p&gt;

&lt;h3 id=&quot;region-proposals&quot;&gt;Region Proposals&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Find “blobby” image regions that are likely to contain objects&lt;/li&gt;
  &lt;li&gt;“Class-agnostic” object detector&lt;/li&gt;
  &lt;li&gt;Look for “blob-like” regions&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;region-proposals-selective-search&quot;&gt;Region Proposals: Selective Search&lt;/h4&gt;

&lt;p&gt;！！！！！！！！！！！！！！&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Uijlings et al, “Selective Search for Object Recognition”, IJCV 2013&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-Selective-Search.png&quot; alt=&quot;Selective Search&quot; /&gt;&lt;/p&gt;

&lt;p&gt;相同的颜色和纹理，然后合并&lt;/p&gt;

&lt;p&gt;还有很多其他的方法：&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-Region-Proposals.png&quot; alt=&quot;Region Proposals&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Hosang et al, “What makes for effective detection proposals?”, PAMI 2015&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;EdgeBoxes&lt;/strong&gt;&lt;/p&gt;

&lt;h2 id=&quot;r-cnn&quot;&gt;R-CNN&lt;/h2&gt;

&lt;p&gt;Girschick et al, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR 2014&lt;/p&gt;

&lt;p&gt;Region-based&lt;/p&gt;

&lt;p&gt;Slide credit: Ross Girschick&lt;/p&gt;

&lt;h4 id=&quot;r-cnn-training&quot;&gt;R-CNN Training&lt;/h4&gt;

&lt;p&gt;Step 1: Train (or download) a classification model for ImageNet (AlexNet)&lt;/p&gt;

&lt;p&gt;Step 2: Fine-tune model for detection&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Instead of 1000 ImageNet classes, want 20 object classes + background&lt;/li&gt;
  &lt;li&gt;Throw away final fully-connected layer, reinitialize from scratch&lt;/li&gt;
  &lt;li&gt;Keep training model using positive / negative regions from detection images&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Step 3: Extract features&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Extract region proposals for all images&lt;/li&gt;
  &lt;li&gt;For each region: warp to CNN input size, run forward through CNN, save pool5 features to disk&lt;/li&gt;
  &lt;li&gt;Have a big hard drive: features are ~200GB for PASCAL dataset!&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Step 4: Train one binary SVM per class to classify region features&lt;/p&gt;

&lt;p&gt;Step 5 (bbox regression): For each class, train a linear regression model to map from cached features to offsets to GT boxes to make up for “slightly wrong” proposals&lt;/p&gt;

&lt;p&gt;Correction&lt;/p&gt;

&lt;hr /&gt;
&lt;p&gt;PASCAL VOC (2010)&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Number of images ~20k，Mean objects per image 2.4&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ImageNet Detection (ILSVRC 2014)&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Number of images ~470k，Mean objects per image 1.1&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;MS-COCO (2014) 更有意思&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Number of images ~120k，Mean objects per image 7.2&lt;/li&gt;
&lt;/ul&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;evaluation&quot;&gt;Evaluation&lt;/h2&gt;

&lt;p&gt;We use a metric called “mean average precision” (mAP)&lt;/p&gt;

&lt;p&gt;0~100，100 is good&lt;/p&gt;

&lt;p&gt;Compute average precision (AP) separately for each class, then average over classes&lt;/p&gt;

&lt;p&gt;A detection is a true positive if it has IoU with a ground-truth box greater than some threshold (usually 0.5) (mAP@0.5)&lt;/p&gt;

&lt;p&gt;Combine all detections from all test images to draw a precision / recall curve for each class; AP is area under the curve&lt;/p&gt;

&lt;p&gt;TL;DR mAP is a number from 0 to 100; high is good&lt;/p&gt;

&lt;p&gt;Wang et al, “Regionlets for Generic Object Detection”, ICCV 2013&lt;/p&gt;

&lt;h2 id=&quot;r-cnn-problems&quot;&gt;R-CNN Problems&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Slow at test-time: need to run full forward pass of CNN for each region proposal&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;SVMs and regressors are post-hoc: CNN features not updated in response to SVMs and regressors&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Complex multistage training pipeline&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;fast-r-cnn&quot;&gt;Fast R-CNN&lt;/h2&gt;

&lt;p&gt;Girschick, “Fast R-CNN”, ICCV 2015&lt;/p&gt;

&lt;p&gt;共用一个 ConvNet&lt;/p&gt;

&lt;p&gt;Get Region from conv feature map，called ‘Rol pooling layer’&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Share computation of convolutional layers between proposals for an image(解决Slow at test-time)&lt;/li&gt;
  &lt;li&gt;Just train the whole system end-to-end all at once!(解决问题2、3)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;region-of-interest-pooling&quot;&gt;Region of Interest Pooling&lt;/h3&gt;

&lt;p&gt;Hi-res conv features: C x H x W with region proposal&lt;/p&gt;

&lt;p&gt;Problem: Fully-connected layers expect low-res conv features: C x h x w&lt;/p&gt;

&lt;p&gt;Max-pool within each grid cell&lt;/p&gt;

&lt;p&gt;R-CNN&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Training Time 84 hours&lt;/li&gt;
  &lt;li&gt;Speed up 1x&lt;/li&gt;
  &lt;li&gt;Test time per image 47 seconds&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Fast R-CNN&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Training Time 9.5 hours&lt;/li&gt;
  &lt;li&gt;Speed up 8.8x&lt;/li&gt;
  &lt;li&gt;Test time per image 0.32 seconds&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Just make the CNN do region proposals too!!!&lt;/p&gt;

&lt;h2 id=&quot;faster-r-cnn&quot;&gt;Faster R-CNN&lt;/h2&gt;

&lt;p&gt;Insert a Region Proposal Network (RPN) after the last convolutional layer!!!!!!!!!!!!!!!!!!&lt;/p&gt;

&lt;p&gt;RPN trained to produce region proposals directly; no need for external region proposals&lt;/p&gt;

&lt;p&gt;After RPN, use RoI Pooling and an upstream classifier and bbox regressor just like Fast R-CNN&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ren et al, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, NIPS 2015&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Slide a small window on the feature map&lt;/p&gt;

&lt;p&gt;Build a small network for:
- classifying object or not-object, and 
- regressing bbox locations&lt;/p&gt;

&lt;p&gt;Position of the sliding window provides localization information with reference to the image&lt;/p&gt;

&lt;p&gt;Box regression provides finer localization information with reference to this sliding window&lt;/p&gt;

&lt;hr /&gt;
&lt;p&gt;使用anchor box，先用预先的几个框试一下
- Use &lt;strong&gt;N anchor boxes&lt;/strong&gt; at each location
- Anchors are translation invariant: use the same ones at every location
- Regression gives offsets from anchor boxes (regressed) anchor shows an object&lt;/p&gt;

&lt;hr /&gt;
&lt;p&gt;Since publication: Joint training! One network, four losses&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;RPN classification (anchor good / bad)&lt;/li&gt;
  &lt;li&gt;RPN regression (anchor -&amp;gt; proposal)&lt;/li&gt;
  &lt;li&gt;Fast R-CNN classification (over classes)&lt;/li&gt;
  &lt;li&gt;Fast R-CNN regression (proposal -&amp;gt; box)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Faster R-CNN&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Test time per image:0.2 seconds&lt;/li&gt;
  &lt;li&gt;(Speedup) 250x&lt;/li&gt;
  &lt;li&gt;mAP (VOC 2007) 66.9&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;spatial transformer network&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;binary linear interpolation&lt;/p&gt;

&lt;h2 id=&quot;object-detection-state-of-the-art&quot;&gt;Object Detection State-of-the-art&lt;/h2&gt;

&lt;p&gt;ResNet 101 + Faster R-CNN + some extras&lt;/p&gt;

&lt;p&gt;He et. al, “Deep Residual Learning for Image Recognition”, arXiv 2015&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24Ensemble.png&quot; alt=&quot;ResNet 101 + Faster R-CNN + some extras&quot; /&gt;&lt;/p&gt;

&lt;h2 id=&quot;yolo-you-only-look-once-detection-as-regression&quot;&gt;YOLO: You Only Look Once Detection as Regression&lt;/h2&gt;

&lt;p&gt;Divide image into S x S grid&lt;/p&gt;

&lt;p&gt;Within each grid cell predict:
B Boxes: 4 coordinates + confidence Class scores: C numbers&lt;/p&gt;

&lt;p&gt;Regression from image to 7 x 7 x (5 * B + C) tensor&lt;/p&gt;

&lt;p&gt;Direct prediction using a CNN&lt;/p&gt;

&lt;p&gt;Redmon et al, “You Only Look Once:Unified, Real-Time Object Detection”, arXiv 2015&lt;/p&gt;

&lt;p&gt;Faster than Faster R-CNN, but not as good&lt;/p&gt;

&lt;h1 id=&quot;code-links&quot;&gt;Code Links&lt;/h1&gt;

&lt;h2 id=&quot;r-cnn-1&quot;&gt;R-CNN&lt;/h2&gt;

&lt;p&gt;(Cafffe + MATLAB): https://github.com/rbgirshick/rcnn Probably don’t use this; too slow&lt;/p&gt;

&lt;h2 id=&quot;fast-r-cnn-1&quot;&gt;Fast R-CNN&lt;/h2&gt;

&lt;p&gt;(Caffe + MATLAB): https://github.com/rbgirshick/fast-rcnn&lt;/p&gt;

&lt;h2 id=&quot;faster-r-cnn-1&quot;&gt;Faster R-CNN&lt;/h2&gt;

&lt;p&gt;(Caffe + MATLAB): https://github.com/ShaoqingRen/faster_rcnn&lt;/p&gt;

&lt;p&gt;(Caffe + Python): https://github.com/rbgirshick/py-faster-rcnn&lt;/p&gt;

&lt;h2 id=&quot;yolo&quot;&gt;YOLO&lt;/h2&gt;

&lt;p&gt;http://pjreddie.com/darknet/yolo/&lt;/p&gt;

&lt;hr /&gt;
&lt;hr /&gt;

&lt;h1 id=&quot;convnets&quot;&gt;ConvNets&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;Visualize patches that maximally activate neurons&lt;/li&gt;
  &lt;li&gt;Visualize the weights&lt;/li&gt;
  &lt;li&gt;Visualize the representation space (e.g. with t-SNE)&lt;/li&gt;
  &lt;li&gt;Occlusion experiments&lt;/li&gt;
  &lt;li&gt;Human experiment comparisons&lt;/li&gt;
  &lt;li&gt;Deconv approaches (single backward pass)&lt;/li&gt;
  &lt;li&gt;Optimization over image approaches (optimization)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;t-sne-visualization&quot;&gt;t-SNE visualization&lt;/h2&gt;

&lt;p&gt;Embed high-dimensional points so that locally, pairwise distances are conserved&lt;/p&gt;

&lt;p&gt;i.e. similar things end up in similar places. dissimilar things end up wherever&lt;/p&gt;

&lt;h2 id=&quot;occlusion-experiments&quot;&gt;Occlusion experiments&lt;/h2&gt;

&lt;p&gt;[Zeiler &amp;amp; Fergus 2013]&lt;/p&gt;

&lt;p&gt;as a function of the position of the square of zeros in the original image&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://lvdmaaten.github.io/tsne/&quot;&gt;t-SNE&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://yosinski.com/deepvis&quot;&gt;Deep Visualization Toolbox&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&quot;deconv-approaches&quot;&gt;Deconv approaches&lt;/h2&gt;

&lt;ol&gt;
  &lt;li&gt;Feed image into net&lt;/li&gt;
  &lt;li&gt;Pick a layer, set the gradient there to be all zero except for one 1 for some neuron of interest&lt;/li&gt;
  &lt;li&gt;Backprop to image(“Guided backpropagation:” instead)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;[Visualizing and Understanding Convolutional Networks, Zeiler and Fergus 2013]&lt;/p&gt;

&lt;p&gt;[Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, Simonyan et al., 2014]&lt;/p&gt;

&lt;p&gt;[Striving for Simplicity: The all convolutional net, Springenberg, Dosovitskiy, et al., 2015]&lt;/p&gt;

&lt;p&gt;Backward pass for a ReLU (will be changed in Guided Backprop)&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-Deconv-approaches.png&quot; alt=&quot;Deconv approches&quot; /&gt;&lt;/p&gt;

&lt;p&gt;guided bp vs backward pass&lt;/p&gt;

&lt;h2 id=&quot;optimization-to-image&quot;&gt;Optimization to Image&lt;/h2&gt;

&lt;p&gt;Q: can we find an image that maximizes some class score?
！！！！！！！！！&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;feed in zeros.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;set the gradient of the scores vector to be [0,0,….1,….,0], then backprop to image&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;do a small “image update”&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;forward the image through the network.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;go back to 2.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;hr /&gt;

&lt;h3 id=&quot;visualize-the-data-gradient&quot;&gt;Visualize the Data gradient:&lt;/h3&gt;

&lt;p&gt;[Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps]&lt;/p&gt;

&lt;p&gt;Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, 2014&lt;/p&gt;

&lt;p&gt;at each pixel take abs val, and max over channels&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use grabcut for segmentation&lt;/strong&gt;&lt;/p&gt;

&lt;hr /&gt;

&lt;ol&gt;
  &lt;li&gt;Forward an image&lt;/li&gt;
  &lt;li&gt;Set activations in layer of interest to all zero, except for a 1.0 for a neuron of interest&lt;/li&gt;
  &lt;li&gt;Backprop to image&lt;/li&gt;
  &lt;li&gt;Do an “image update”&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Proposed a different form of regularizing the image&lt;/p&gt;

&lt;p&gt;Repeat:
- Update the image x with gradient from some unit of interest
- Blur x a bit
- Take any pixel with small norm to zero (to encourage sparsity)&lt;/p&gt;

&lt;h3 id=&quot;given-a-cnn-code-is-it-possible-to-reconstruct-the-original-image&quot;&gt;Given a CNN code, is it possible to reconstruct the original image?&lt;/h3&gt;

&lt;p&gt;Reconstructions from the representation after last last pooling layer(immediately before the first Fully Connected layer)&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-pooling-layer-befor-fc.png&quot; alt=&quot;Reconstructions from the representation after last last pooling layer&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Reconstructions from intermediate layers&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-24-intermediate-layers.png&quot; alt=&quot;intermediate layers&quot; /&gt;&lt;/p&gt;

&lt;h2 id=&quot;deep-dream&quot;&gt;Deep Dream&lt;/h2&gt;

&lt;p&gt;DeepDream modifies the image in a way that “boosts” all activations, at any layer&lt;/p&gt;

&lt;p&gt;对某个neuron刺激最强的那个==&lt;/p&gt;

&lt;h2 id=&quot;neural-style&quot;&gt;Neural Style&lt;/h2&gt;

&lt;p&gt;Step 1: Extract content targets (ConvNet activations of all layers for the given content image)&lt;/p&gt;

&lt;p&gt;Step 2: Extract style targets (Gram matrices of ConvNet activations of all layers for the given style image)&lt;/p&gt;

&lt;p&gt;Step 3: Optimize over image to have:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The content of the content image (activations match content)&lt;/li&gt;
  &lt;li&gt;The style of the style image (Gram matrices of activations match style)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;question-can-we-use-this-to-fool-convnets&quot;&gt;Question: Can we use this to “fool” ConvNets?&lt;/h2&gt;

&lt;p&gt;[Intriguing properties of neural networks, Szegedy et al., 2013]&lt;/p&gt;

&lt;p&gt;[Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images Nguyen, Yosinski, Clune, 2014]&lt;/p&gt;

&lt;p&gt;“primary cause of neural networks’ vulnerability to adversarial perturbation is their linear nature“&lt;/p&gt;

&lt;p&gt;EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES
[Goodfellow, Shlens &amp;amp; Szegedy, 2014]&lt;/p&gt;

&lt;p&gt;In particular, this is not a problem with Deep Learning, and has little to do with ConvNets specifically. Same issue would come up with Neural Nets in any other modalities.&lt;/p&gt;

&lt;p&gt;反向加入然后拒绝&lt;/p&gt;

&lt;p&gt;Backpropping to the image is powerful. It can be used for:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Understanding (e.g. visualize optimal stimuli for arbitrary neurons)&lt;/li&gt;
  &lt;li&gt;Segmenting objects in the image (kind of)&lt;/li&gt;
  &lt;li&gt;Inverting codes and introducing privacy concerns&lt;/li&gt;
  &lt;li&gt;Fun (NeuralStyle/DeepDream)&lt;/li&gt;
  &lt;li&gt;Confusion and chaos (Adversarial examples)&lt;/li&gt;
&lt;/ul&gt;

</description>
        <pubDate>Thu, 24 Mar 2016 11:04:43 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/24/lecture-8-localization--detection-stanford-cs231n/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/24/lecture-8-localization--detection-stanford-cs231n/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>卷积神经网络正式内容笔记 Stanford CS231n 2016</title>
        <description>&lt;h2 id=&quot;convnets&quot;&gt;ConvNets&lt;/h2&gt;

&lt;h2 id=&quot;mathematically&quot;&gt;Mathematically&lt;/h2&gt;

&lt;p&gt;32&lt;em&gt;32&lt;/em&gt;3(3 channels) Image&lt;/p&gt;

&lt;p&gt;filter:5&lt;em&gt;5&lt;/em&gt;3 filter 
- &lt;strong&gt;Convolve&lt;/strong&gt; the filter with the image: Slide over the image spatially&lt;/p&gt;

&lt;p&gt;activation map =&amp;gt; 28 * 28 * 1，使用的是$w^T x +b$&lt;/p&gt;

&lt;p&gt;then,there is a second filter(5&lt;em&gt;5&lt;/em&gt;3).then there is another activation map.&lt;/p&gt;

&lt;p&gt;if we had 6 5x5 filters, we’ll get 6 separate activation maps.&lt;/p&gt;

&lt;p&gt;stack up to get 28&lt;em&gt;28&lt;/em&gt;6 new image,This operation name convolution.&lt;/p&gt;

&lt;h3 id=&quot;conv-relu&quot;&gt;Conv ReLU&lt;/h3&gt;

&lt;p&gt;Conv -&amp;gt; ReLU -&amp;gt; Pool -&amp;gt; Conv -&amp;gt; ReLU -&amp;gt; Pool -&amp;gt; …-&amp;gt; Pool -&amp;gt;FC -&amp;gt; Output&lt;/p&gt;

&lt;h3 id=&quot;common-to-zero-pad-the-border&quot;&gt;Common to zero pad the border&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;边界加上一圈border ，0 Pixel&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;一般stride 1&lt;/li&gt;
  &lt;li&gt;Conv每层都在减少（减小的太快不好）所以要加padding，加边界&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;eg:32x32 input convolved repeatedly with 5x5 filters shrinks volumes spatially! (32 -&amp;gt; 28 -&amp;gt; 24 …). Shrinking too fast is not good, doesn’t work well.&lt;/p&gt;

&lt;p&gt;Each filter has 5&lt;em&gt;5&lt;/em&gt;3 + 1 = 76 params (+1 for bias)&lt;/p&gt;

&lt;p&gt;76*10&lt;/p&gt;

&lt;h3 id=&quot;summary&quot;&gt;Summary&lt;/h3&gt;

&lt;p&gt;输入长宽深度：$W_1 \times H_1 \times D_1$&lt;/p&gt;

&lt;p&gt;4个参数：
- K filter的个数 number of filters ，K是2的次幂powers of 2 eg 32,64,128
- F 空间范围 spatial extent filter的空间范围
- S 步长 stride
- P 0 padding的总数&lt;/p&gt;

&lt;p&gt;Common settings:&lt;/p&gt;

&lt;p&gt;K = (powers of 2, e.g. 32, 64, 128, 512)
- F = 3, S = 1, P = 1
- F = 5, S = 1, P = 2
- F = 5, S = 2, P = ? (whatever fits)
- F = 1, S = 1, P = 0&lt;/p&gt;

&lt;p&gt;1x1 的过滤，为了深度&lt;/p&gt;

&lt;p&gt;F一般都是odd，奇数~&lt;/p&gt;

&lt;p&gt;默认都处理正方形&lt;/p&gt;

&lt;p&gt;一般不会去在一层里用多种大小的Filter&lt;/p&gt;

&lt;p&gt;计算得到第二层的长宽深度：
&lt;script type=&quot;math/tex&quot;&gt;W_2 = (W_1 - F + 2 P) / S + 1
H_2 = (H_1 - F + 2 P) / S + 1
D_2 = K&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;因为参数共享，每个过滤器有$F \cdot F \cdot D_1$这么多weights参数，一共有$(F \cdot F \cdot D_1) \cdot K$ + K biases&lt;/p&gt;

&lt;p&gt;输出一共有d-th层$ W_2 \times H_2$的卷积结果&lt;/p&gt;

&lt;h2 id=&quot;conv-layerneuron-view&quot;&gt;把Conv Layer变成Neuron View&lt;/h2&gt;

&lt;p&gt;An activation map is a 28x28 sheet of neuron outputs:
1. Each is connected to a small region in the input
2. &lt;strong&gt;All of them share parameters&lt;/strong&gt; 因为都是用一个filter变过来的&lt;/p&gt;

&lt;p&gt;“5x5 filter” -&amp;gt; “5x5 receptive field for each neuron”&lt;/p&gt;

&lt;h2 id=&quot;pool--fc&quot;&gt;Pool / FC&lt;/h2&gt;

&lt;h3 id=&quot;pooling-layers&quot;&gt;Pooling Layers&lt;/h3&gt;

&lt;p&gt;做 downsampling，例如从224&lt;em&gt;224&lt;/em&gt;64-&amp;gt;112&lt;em&gt;112&lt;/em&gt;64&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;makes the representations smaller and more manageable&lt;/li&gt;
  &lt;li&gt;operates over each activation map independently:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;downsampling最常用的是 max pool 用一个2x2的filter，步长为2来求max&lt;/p&gt;

&lt;p&gt;Pooling不改变 Depth&lt;/p&gt;

&lt;h3 id=&quot;fully-connected-layer-fc&quot;&gt;Fully Connected Layer （FC）&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Contains neurons that connect to the entire input volume, as in ordinary Neural Networks&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html&quot;&gt;JavaScript Demo&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;可视化很困难&lt;/p&gt;

&lt;h3 id=&quot;case-study&quot;&gt;Case Study&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;LeCUN 1998&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Conv-&amp;gt;SubSampleing-&amp;gt;Conv-&amp;gt;Sub-&amp;gt;FC-&amp;gt;FC-&amp;gt;GC&lt;/p&gt;

&lt;h2 id=&quot;alexnet&quot;&gt;AlexNet&lt;/h2&gt;

&lt;p&gt;8 layers&lt;/p&gt;

&lt;p&gt;input : 227&lt;em&gt;227&lt;/em&gt;3 images&lt;/p&gt;

&lt;p&gt;当时GPU Mem不够，所以用了两条分离的线（不考虑）&lt;/p&gt;

&lt;p&gt;First Layer： 96 11*11 filters&lt;/p&gt;

&lt;p&gt;=&amp;gt; Output: 55&lt;em&gt;55&lt;/em&gt;96&lt;/p&gt;

&lt;p&gt;Parameters: 11&lt;em&gt;11&lt;/em&gt;3 *96 = 35K&lt;/p&gt;

&lt;p&gt;Second Layer: POOL1 : 3x3 at stride 2.No paramter&lt;/p&gt;

&lt;p&gt;Pool layer是没有parameters的&lt;/p&gt;

&lt;p&gt;Details/Retrospectives:
- first use of ReLU
- used Norm layers (not common anymore) - heavy data augmentation
- dropout 0.5  最后几层FC
- batch size 128
- SGD Momentum 0.9
- Learning rate 1e-2, reduced by 10 manually when val accuracy plateaus
- L2 weight decay 5e-4
- 7 CNN ensemble: 18.2% -&amp;gt; 15.4%&lt;/p&gt;

&lt;p&gt;FC7 feature，再分类之前的最后一次FC&lt;/p&gt;

&lt;h2 id=&quot;zfnet&quot;&gt;ZFNet&lt;/h2&gt;

&lt;p&gt;8 layers&lt;/p&gt;

&lt;p&gt;[Zeiler and Fergus, 2013]&lt;/p&gt;

&lt;p&gt;AlexNet but:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;CONV1: change from (11x11 stride 4) to (7x7 stride 2)&lt;/li&gt;
  &lt;li&gt;CONV3,4,5: instead of 384, 384, 256 filters use 512, 1024, 512&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ImageNet top 5 error: 15.4% -&amp;gt; 14.8%&lt;/p&gt;

&lt;h2 id=&quot;vggnet&quot;&gt;VGGNet&lt;/h2&gt;

&lt;p&gt;19 layers&lt;/p&gt;

&lt;p&gt;[Simonyan and Zisserman, 2014]&lt;/p&gt;

&lt;p&gt;Only 3x3 CONV stride 1, pad 1 and 2x2 MAX POOL stride 2&lt;/p&gt;

&lt;p&gt;19层，11.2% top 5 error in ILSVRC 2013 -&amp;gt;
7.3% top 5 error&lt;/p&gt;

&lt;p&gt;主要params在FC层&lt;/p&gt;

&lt;h2 id=&quot;googlenet&quot;&gt;GoogLeNet&lt;/h2&gt;

&lt;p&gt;22 layers&lt;/p&gt;

&lt;p&gt;[Szegedy et al., 2014]&lt;/p&gt;

&lt;p&gt;Inception Module&lt;/p&gt;

&lt;p&gt;Inception Layers ，Average&lt;/p&gt;

&lt;p&gt;reduce computation &amp;amp; memory&lt;/p&gt;

&lt;h2 id=&quot;resnet&quot;&gt;ResNet&lt;/h2&gt;

&lt;p&gt;152 layers [He et al., 2015]&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;并不知道何时converge…可能仅仅是训练了两周了，累了….&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;ILSVRC 2015 winner (3.6% top 5 error)&lt;/p&gt;

&lt;p&gt;plain net vs res net&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;2-3 weeks of training on 8 GPU machine&lt;/li&gt;
  &lt;li&gt;at runtime: &lt;strong&gt;faster&lt;/strong&gt; than a VGGNet! (even though it has 8x more layers)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Skip connections&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-23-res-vs-plain.png&quot; alt=&quot;res net vs plain net&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/2016-03-23-resnet.png&quot; alt=&quot;res net vs plain net&quot; /&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Batch Normalization after every CONV layer&lt;/li&gt;
  &lt;li&gt;Xavier/2 initialization from He et al.&lt;/li&gt;
  &lt;li&gt;SGD + Momentum (0.9)&lt;/li&gt;
  &lt;li&gt;Learning rate: 0.1, divided by 10 when validation error plateaus&lt;/li&gt;
  &lt;li&gt;Mini-batch size 256&lt;/li&gt;
  &lt;li&gt;Weight decay of 1e-5&lt;/li&gt;
  &lt;li&gt;No dropout used&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;alpha-go&quot;&gt;Alpha Go&lt;/h2&gt;

&lt;p&gt;policy network&lt;/p&gt;

&lt;p&gt;[19x19x48] Input
- CONV1: 192 5x5 filters , stride 1, pad 2 =&amp;gt; [19x19x192]
- CONV2..12: 192 3x3 filters, stride 1, pad 1 =&amp;gt; [19x19x192]
- CONV: 1 1x1 filter, stride 1, pad 0 =&amp;gt; [19x19] =&amp;gt; probability map of promising moves&lt;/p&gt;

</description>
        <pubDate>Mon, 21 Mar 2016 21:25:34 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/21/lecture-7-convnets-stanford-cs231n-2016-notes/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/21/lecture-7-convnets-stanford-cs231n-2016-notes/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
      <item>
        <title>MathJax and Latex syntax</title>
        <description>&lt;p&gt;本文转载自&lt;a href=&quot;http://mlworks.cn/posts/introduction-to-mathjax-and-latex-expression/&quot;&gt;Mathjax与LaTex公式简介,Ryan Zhao&lt;/a&gt;。&lt;/p&gt;

&lt;p&gt;本文从&lt;a href=&quot;http://math.stackexchange.com&quot;&gt;math.stackexchange.com&lt;/a&gt;上名为&lt;a href=&quot;http://meta.math.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference/5044&quot;&gt;MathJax basic tutorial and quick reference&lt;/a&gt;的问题翻译而来，并有所改动。主要讲述了如何使用MathJax和相关的Latex语法。&lt;/p&gt;

&lt;h2 id=&quot;mathjax&quot;&gt;MathJax简介&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;http://www.mathjax.org/&quot; title=&quot;MathJax官方网站&quot;&gt;MathJax&lt;/a&gt;是一款运行在浏览器中的开源的数学符号渲染引擎，使用MathJax可以方便的在浏览器中显示数学公式，不需要使用图片。目前，MathJax可以解析Latex、MathML和ASCIIMathML的标记语言。
MathJax项目于2009年开始，发起人有American Mathematical Society, Design Science等，还有众多的支持者，个人感觉MathJax会成为今后数学符号渲染引擎中的主流，也许现在已经是了。
本文接下来会讲述MathJax的基础用法，但不涉及MathJax的安装及配置。此外，推荐使用&lt;a href=&quot;https://stackedit.io/&quot;&gt;StackEdit&lt;/a&gt;学习MathJax的语法，它支持Markdown和MathJax，本文使用此编辑器撰写。&lt;/p&gt;

&lt;!--more--&gt;
&lt;p&gt;## 基础&lt;/p&gt;

&lt;h3 id=&quot;section&quot;&gt;公式标记与查看公式&lt;/h3&gt;
&lt;p&gt;使用MathJax时，需要用一些适当的标记告诉MathJax某段文本是公式代码。此外，MathJax中的公式排版有两种方式，inline和displayed。inline表示公式嵌入到文本段中，displayed表示公式独自成为一个段落。例如，$f(x)=3 \times x$这是一个inline公式，而下面&lt;script type=&quot;math/tex&quot;&gt;f(x)=3 \times x&lt;/script&gt;是一个displayed公式。&lt;/p&gt;

&lt;p&gt;在MathJax中，默认的displayed公式分隔符有&lt;code&gt;$$...$$&lt;/code&gt; 和&lt;code&gt;\\[...\\]&lt;/code&gt;,而默认的inline公式分隔符为&lt;code&gt; \(...\) &lt;/code&gt;,当然这些都是可以自定义的，具体配置请参考&lt;a href=&quot;http://docs.mathjax.org/en/latest/start.html#tex-and-latex-input&quot;&gt;文档&lt;/a&gt;。下文中，使用&lt;code&gt;$$...$$&lt;/code&gt;作为displayed分隔符，&lt;code&gt;$...$&lt;/code&gt;作为inline分隔符。&lt;/p&gt;

&lt;p&gt;此外，可以在渲染完成的公式上方右键点击，唤出右键菜单。在菜单中提供了查看公式代码、设置显示效果和渲染模式的选项。&lt;/p&gt;

&lt;h3 id=&quot;section-1&quot;&gt;希腊字母&lt;/h3&gt;
&lt;p&gt;请参见下表：&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;名称&lt;/th&gt;
      &lt;th&gt;大写&lt;/th&gt;
      &lt;th&gt;Tex&lt;/th&gt;
      &lt;th&gt;小写&lt;/th&gt;
      &lt;th&gt;Tex&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;alpha&lt;/td&gt;
      &lt;td&gt;$A$&lt;/td&gt;
      &lt;td&gt;A&lt;/td&gt;
      &lt;td&gt;$\alpha$&lt;/td&gt;
      &lt;td&gt;\alpha&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;beta&lt;/td&gt;
      &lt;td&gt;$B$&lt;/td&gt;
      &lt;td&gt;B&lt;/td&gt;
      &lt;td&gt;$\beta$&lt;/td&gt;
      &lt;td&gt;\beta&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;gamma&lt;/td&gt;
      &lt;td&gt;$\Gamma$&lt;/td&gt;
      &lt;td&gt;\Gamma&lt;/td&gt;
      &lt;td&gt;$\gamma$&lt;/td&gt;
      &lt;td&gt;\gamma&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;delta&lt;/td&gt;
      &lt;td&gt;$\Delta$&lt;/td&gt;
      &lt;td&gt;\Delta&lt;/td&gt;
      &lt;td&gt;$\delta$&lt;/td&gt;
      &lt;td&gt;\delta&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;epsilon&lt;/td&gt;
      &lt;td&gt;$E$&lt;/td&gt;
      &lt;td&gt;E&lt;/td&gt;
      &lt;td&gt;$\epsilon$&lt;/td&gt;
      &lt;td&gt;\epsilon&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;zeta&lt;/td&gt;
      &lt;td&gt;$Z$&lt;/td&gt;
      &lt;td&gt;Z&lt;/td&gt;
      &lt;td&gt;$\zeta$&lt;/td&gt;
      &lt;td&gt;\zeta&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;eta&lt;/td&gt;
      &lt;td&gt;$H$&lt;/td&gt;
      &lt;td&gt;H&lt;/td&gt;
      &lt;td&gt;$\eta$&lt;/td&gt;
      &lt;td&gt;\eta&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;theta&lt;/td&gt;
      &lt;td&gt;$\Theta$&lt;/td&gt;
      &lt;td&gt;\Theta&lt;/td&gt;
      &lt;td&gt;$\theta$&lt;/td&gt;
      &lt;td&gt;\theta&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;iota&lt;/td&gt;
      &lt;td&gt;$I$&lt;/td&gt;
      &lt;td&gt;I&lt;/td&gt;
      &lt;td&gt;$\iota$&lt;/td&gt;
      &lt;td&gt;\iota&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;kappa&lt;/td&gt;
      &lt;td&gt;$K$&lt;/td&gt;
      &lt;td&gt;K&lt;/td&gt;
      &lt;td&gt;$\kappa$&lt;/td&gt;
      &lt;td&gt;\kappa&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;lambda&lt;/td&gt;
      &lt;td&gt;$\Lambda$&lt;/td&gt;
      &lt;td&gt;\Lambda&lt;/td&gt;
      &lt;td&gt;$\lambda$&lt;/td&gt;
      &lt;td&gt;\lambda&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;mu&lt;/td&gt;
      &lt;td&gt;$M$&lt;/td&gt;
      &lt;td&gt;M&lt;/td&gt;
      &lt;td&gt;$\mu$&lt;/td&gt;
      &lt;td&gt;\mu&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;nu&lt;/td&gt;
      &lt;td&gt;$N$&lt;/td&gt;
      &lt;td&gt;N&lt;/td&gt;
      &lt;td&gt;$\nu$&lt;/td&gt;
      &lt;td&gt;\nu&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;xi&lt;/td&gt;
      &lt;td&gt;$\Xi$&lt;/td&gt;
      &lt;td&gt;\Xi&lt;/td&gt;
      &lt;td&gt;$\xi$&lt;/td&gt;
      &lt;td&gt;\xi&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;omicron&lt;/td&gt;
      &lt;td&gt;$O$&lt;/td&gt;
      &lt;td&gt;O&lt;/td&gt;
      &lt;td&gt;$\omicron$&lt;/td&gt;
      &lt;td&gt;\omicron&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;pi&lt;/td&gt;
      &lt;td&gt;$\Pi$&lt;/td&gt;
      &lt;td&gt;\Pi&lt;/td&gt;
      &lt;td&gt;$\pi$&lt;/td&gt;
      &lt;td&gt;\pi&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;rho&lt;/td&gt;
      &lt;td&gt;$P$&lt;/td&gt;
      &lt;td&gt;P&lt;/td&gt;
      &lt;td&gt;$\rho$&lt;/td&gt;
      &lt;td&gt;\rho&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;sigma&lt;/td&gt;
      &lt;td&gt;$\Sigma$&lt;/td&gt;
      &lt;td&gt;\Sigma&lt;/td&gt;
      &lt;td&gt;$\sigma$&lt;/td&gt;
      &lt;td&gt;\sigma&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;tau&lt;/td&gt;
      &lt;td&gt;$T$&lt;/td&gt;
      &lt;td&gt;T&lt;/td&gt;
      &lt;td&gt;$\tau$&lt;/td&gt;
      &lt;td&gt;\tau&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;upsilon&lt;/td&gt;
      &lt;td&gt;$\Upsilon$&lt;/td&gt;
      &lt;td&gt;\Upsilon&lt;/td&gt;
      &lt;td&gt;$\upsilon$&lt;/td&gt;
      &lt;td&gt;\upsilon&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;phi&lt;/td&gt;
      &lt;td&gt;$\Phi$&lt;/td&gt;
      &lt;td&gt;\Phi&lt;/td&gt;
      &lt;td&gt;$\phi$&lt;/td&gt;
      &lt;td&gt;\phi&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;chi&lt;/td&gt;
      &lt;td&gt;$X$&lt;/td&gt;
      &lt;td&gt;X&lt;/td&gt;
      &lt;td&gt;$\chi$&lt;/td&gt;
      &lt;td&gt;\chi&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;psi&lt;/td&gt;
      &lt;td&gt;$\Psi$&lt;/td&gt;
      &lt;td&gt;\Psi&lt;/td&gt;
      &lt;td&gt;$\psi$&lt;/td&gt;
      &lt;td&gt;\psi&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;omega&lt;/td&gt;
      &lt;td&gt;$\Omega $&lt;/td&gt;
      &lt;td&gt;\Omega&lt;/td&gt;
      &lt;td&gt;$\omega$&lt;/td&gt;
      &lt;td&gt;\omega&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;h3 id=&quot;section-2&quot;&gt;上标与下标&lt;/h3&gt;
&lt;p&gt;上标和下标分别使用^与_，例如x_i^2：$x_i^2$。默认情况下，上下标符号仅仅对下一个组起作用。一个组即单个字符或者使用{..}包裹起来的内容。也就是说，如果使用10^10，会得到$10^10$，而10^{10}才是$10^{10}$。同时，大括号还能消除二义性，如x^5^6将得到一个错误，必须使用大括号来界定^的结合性，如{x^5}^6：${x^5}^6$ 或者 x^{5^6}：$x^{5^6}$。&lt;/p&gt;

&lt;h3 id=&quot;section-3&quot;&gt;括号&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;小括号与方括号：使用原始的( )，[ ]即可，如(2+3)[4+4]：$(2+3)[4+4]$&lt;/li&gt;
  &lt;li&gt;大括号：时由于大括号{}被用来分组，因此需要使用\{和\}表示大括号，也可以使用\lbrace 和\rbrace来表示。如\{a&lt;em&gt;b\}:${a&lt;/em&gt;b}$，\lbrace a&lt;em&gt;b \rbrace：$\lbrace a&lt;/em&gt;b \rbrace$。&lt;/li&gt;
  &lt;li&gt;尖括号：使用\langle 和 \rangle表示左尖括号和右尖括号。如\langle x \rangle：$\langle x \rangle$。&lt;/li&gt;
  &lt;li&gt;上取整：使用\lceil 和 \rceil 表示。 如，\lceil x \rceil：$\lceil x \rceil$。&lt;/li&gt;
  &lt;li&gt;下取整：使用\lfloor 和 \rfloor 表示。如，\lfloor x \rfloor：$\lfloor x \rfloor$。&lt;/li&gt;
  &lt;li&gt;不可见括号：使用.表示。&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;需要注意的是，原始符号并不会随着公式大小缩放。如，(\frac12)：$(\frac12)$。可以使用\left(…\right)来自适应的调整括号大小。如下，
&lt;script type=&quot;math/tex&quot;&gt;\lbrace\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}\rbrace\tag{1.1}&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;&lt;script type=&quot;math/tex&quot;&gt;\left \lbrace \sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6} \right\rbrace\tag{1.2}&lt;/script&gt;
可以看到，公式1.2中的括号是经过缩放的。&lt;/p&gt;

&lt;h3 id=&quot;section-4&quot;&gt;求和与积分&lt;/h3&gt;
&lt;p&gt;\sum用来表示求和符号，其下标表示求和下限，上标表示上限。如，\sum_1^n：$\sum_1^n$。&lt;/p&gt;

&lt;p&gt;\int用来表示积分符号，同样地，其上下标表示积分的上下限。如，\int_1^\infty：$\int_1^\infty$。&lt;/p&gt;

&lt;p&gt;与此类似的符号还有，\prod：$\prod$，\bigcup:$\bigcup$，\bigcap：$\bigcap$，\iint：$\iint$。&lt;/p&gt;

&lt;h3 id=&quot;section-5&quot;&gt;分式与根式&lt;/h3&gt;
&lt;p&gt;分式的表示。第一种，使用\frac ab，\frac作用于其后的两个组a，b，结果为$\frac ab$。如果你的分子或分母不是单个字符，请使用{..}来分组。第二种，使用\over来分隔一个组的前后两部分，如{a+1\over b+1}：${a+1\over b+1}$。&lt;/p&gt;

&lt;p&gt;根式使用\sqrt来表示。如，\sqrt[4]{\frac xy} ：$\sqrt[4]{\frac xy} $&lt;/p&gt;

&lt;h3 id=&quot;section-6&quot;&gt;字体&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;使用\mathbb或\Bbb显示黑板粗体字，此字体经常用来表示代表实数、整数、有理数、复数的大写字母。如，$\mathbb{CHNQRZ}$。&lt;/li&gt;
  &lt;li&gt;使用\mathbf显示黑体字，如，$\mathbf{ABCDEFGHIJKLMNOPQRSTUVWXYZ}$，$\mathbf{abcdefghijklmnopqrstuvwxyz}$。&lt;/li&gt;
  &lt;li&gt;使用\mathtt显示打印机字体，如，$\mathtt{ABCDEFGHIJKLMNOPQRSTUVWXYZ}$，$\mathtt{abcdefghijklmnopqrstuvwxyz}$。&lt;/li&gt;
  &lt;li&gt;使用\mathrm显示罗马字体，如，$\mathrm{ABCDEFGHIJKLMNOPQRSTUVWXYZ}$，$\mathrm{abcdefghijklmnopqrstuvwxyz}$。&lt;/li&gt;
  &lt;li&gt;使用\mathscr显示手写体，如，$\mathscr{ABCDEFGHIJKLMNOPQRSTUVWXYZ}$。&lt;/li&gt;
  &lt;li&gt;使用\mathfrak显示Fraktur字母（一种德国字体），如$\mathfrak{ABCDEFGHIJKLMNOPQRSTUVWXYZ}$，$\mathrm{abcdefghijklmnopqrstuvwxyz}$。&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;section-7&quot;&gt;特殊函数与符号&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;常见的三角函数，求极限符号可直接使用+缩写即可。如$\sin x$,$\arctan x$,$\lim_{1\to\infty}$。&lt;/li&gt;
  &lt;li&gt;比较运算符：\lt \gt \le \ge \neq ： $\lt \gt \le \ge \neq$。可以在这些运算符前面加上\not，如\not\lt：$\not\lt$。&lt;/li&gt;
  &lt;li&gt;\times \div \pm \mp表示：$\times \div \pm \mp$，\cdot表示居中的点，x \cdot y : $x \cdot y$。&lt;/li&gt;
  &lt;li&gt;集合关系与运算：\cup \cap \setminus \subset \subseteq \subsetneq \supset \in \notin \emptyset \varnothing ：$\cup \cap \setminus \subset \subseteq \subsetneq \supset \in \notin \emptyset \varnothing$.&lt;/li&gt;
  &lt;li&gt;表示排列使用{n+1 \choose 2k} 或 \binom{n+1}{2k}，${n+1 \choose 2k}$。&lt;/li&gt;
  &lt;li&gt;箭头：\to \rightarrow \leftarrow \Rightarrow \Leftarrow \mapsto : $\to \rightarrow \leftarrow \Rightarrow \Leftarrow \mapsto$。&lt;/li&gt;
  &lt;li&gt;逻辑运算符：\land \lor \lnot \forall \exists \top \bot \vdash \vDash ： $\land \lor \lnot \forall \exists \top \bot \vdash \vDash$。&lt;/li&gt;
  &lt;li&gt;\star \ast \oplus \circ \bullet ： $\star \ast \oplus \circ \bullet$。&lt;/li&gt;
  &lt;li&gt;\approx \sim \cong \equiv \prec ： $\approx \sim \cong \equiv \prec$。&lt;/li&gt;
  &lt;li&gt;\infty \aleph_0 $\infty  \aleph_0$ \nabla \partial $\nabla \partial$ \Im \Re $Im \Re$。&lt;/li&gt;
  &lt;li&gt;模运算 \pmode, 如，a\equiv b\pmod n：$a\equiv b\pmod n$。&lt;/li&gt;
  &lt;li&gt;\ldots与\cdots，其区别是dots的位置不同，ldots位置稍低，cdots位置居中。$a_1+a_2+\cdots+a_n$，$a_1, a_2, \ldots ,a_n$。&lt;/li&gt;
  &lt;li&gt;一些希腊字母具有变体形式，如 \epsilon \varepsilon : $ \epsilon \varepsilon$, \phi \varphi $\phi \varphi$。&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;使用&lt;a href=&quot;http://detexify.kirelabs.org/classify.html&quot;&gt;Detexify&lt;/a&gt;，你可以在网页上画出符号，Detexify会给出相似的符号及其代码。这是一个方便的功能，但是不能保证它给出的符号可以在MathJax中使用，你可以参考&lt;a href=&quot;http://docs.mathjax.org/en/latest/tex.html#supported-latex-commands&quot;&gt;supported-latex-commands&lt;/a&gt;确定MathJax是否支持此符号。&lt;/p&gt;

&lt;h3 id=&quot;section-8&quot;&gt;空间&lt;/h3&gt;
&lt;p&gt;通常MathJax通过内部策略自己管理公式内部的空间，因此a…b与a…….b（.表示空格）都会显示为$ab$。可以通过在ab间加入\,增加些许间隙，\;增加较宽的间隙，\quad 与 \qquad 会增加更大的间隙，如，$a\qquad b$。&lt;/p&gt;

&lt;h3 id=&quot;section-9&quot;&gt;顶部符号&lt;/h3&gt;
&lt;p&gt;对于单字符，\hat：$\hat x$，多字符可以使用\widehat,$\widehat {xy}$.类似的还有\hat,\overline,\vec,\overrightarrow, \dot \ddot : $\hat x \quad \overline {xyz} \quad \vec  a \quad \overrightarrow {x} \quad \dot x \quad \ddot x $。&lt;/p&gt;

&lt;h3 id=&quot;section-10&quot;&gt;结束&lt;/h3&gt;
&lt;p&gt;基础部分就是这些。需要注意的是一些MathJax使用的特殊字符，可以使用\转义为原来的含义。如&lt;code&gt;\$&lt;/code&gt;表示‘$‘，&lt;code&gt;\\_&lt;/code&gt;表示下划线。&lt;/p&gt;

&lt;h2 id=&quot;section-11&quot;&gt;表格&lt;/h2&gt;
&lt;p&gt;使用&lt;code&gt;$$\begin{array}{列样式}...\end{array}$$&lt;/code&gt;这样的形式来创建表格，列样式可以是clr表示居中，左，右对齐，还可以使用|表示一条竖线。表格中 各行使用\\分隔，各列使用&amp;amp;分隔。使用\hline在本行前加入一条直线。
例如，
&lt;code&gt;
$$
\begin{array}{c|lcr}
n &amp;amp; \text{Left} &amp;amp; \text{Center} &amp;amp; \text{Right} \\
\hline
1 &amp;amp; 0.24 &amp;amp; 1 &amp;amp; 125 \\
2 &amp;amp; -1 &amp;amp; 189 &amp;amp; -8 \\
3 &amp;amp; -20 &amp;amp; 2000 &amp;amp; 1+10i \\
\end{array}
$$
&lt;/code&gt;
结果：&lt;/p&gt;

&lt;div&gt;
$$
\begin{array}{c|lcr}
n &amp;amp; \text{Left} &amp;amp; \text{Center} &amp;amp; \text{Right} \\
\hline
1 &amp;amp; 0.24 &amp;amp; 1 &amp;amp; 125 \\
2 &amp;amp; -1 &amp;amp; 189 &amp;amp; -8 \\
3 &amp;amp; -20 &amp;amp; 2000 &amp;amp; 1+10i \\
\end{array}
$$
&lt;/div&gt;

&lt;p&gt;一个复杂的例子如下：&lt;/p&gt;

&lt;div&gt;
$$
% outer vertical array of arrays
\begin{array}{c}
% inner horizontal array of arrays
\begin{array}{cc}
% inner array of minimum values
\begin{array}{c|cccc}
\text{min} &amp;amp; 0 &amp;amp; 1 &amp;amp; 2 &amp;amp; 3\\
\hline
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0\\
1 &amp;amp; 0 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1\\
2 &amp;amp; 0 &amp;amp; 1 &amp;amp; 2 &amp;amp; 2\\
3 &amp;amp; 0 &amp;amp; 1 &amp;amp; 2 &amp;amp; 3
\end{array}
&amp;amp;
% inner array of maximum values
\begin{array}{c|cccc}
\text{max}&amp;amp;0&amp;amp;1&amp;amp;2&amp;amp;3\\
\hline
0 &amp;amp; 0 &amp;amp; 1 &amp;amp; 2 &amp;amp; 3\\
1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 2 &amp;amp; 3\\
2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 3\\
3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3
\end{array}
\end{array}
\\
% inner array of delta values
\begin{array}{c|cccc}
\Delta&amp;amp;0&amp;amp;1&amp;amp;2&amp;amp;3\\
\hline
0 &amp;amp; 0 &amp;amp; 1 &amp;amp; 2 &amp;amp; 3\\
1 &amp;amp; 1 &amp;amp; 0 &amp;amp; 1 &amp;amp; 2\\
2 &amp;amp; 2 &amp;amp; 1 &amp;amp; 0 &amp;amp; 1\\
3 &amp;amp; 3 &amp;amp; 2 &amp;amp; 1 &amp;amp; 0
\end{array}
\end{array}
$$
&lt;/div&gt;

&lt;h2 id=&quot;section-12&quot;&gt;矩阵&lt;/h2&gt;

&lt;h3 id=&quot;section-13&quot;&gt;基本用法&lt;/h3&gt;
&lt;p&gt;使用&lt;code&gt;$$\begin{matrix}…\end{matrix}$$&lt;/code&gt;这样的形式来表示矩阵，在\begin与\end之间加入矩阵中的元素即可。矩阵的行之间使用\\分隔，列之间使用&amp;amp;分隔。&lt;/p&gt;

&lt;p&gt;例如
&lt;code&gt;
$$
        \begin{matrix}
        1 &amp;amp; x &amp;amp; x^2 \\
        1 &amp;amp; y &amp;amp; y^2 \\
        1 &amp;amp; z &amp;amp; z^2 \\
        \end{matrix}
$$
&lt;/code&gt;
结果：&lt;/p&gt;

&lt;div&gt;
$$
        \begin{matrix}
        1 &amp;amp; x &amp;amp; x^2 \\
        1 &amp;amp; y &amp;amp; y^2 \\
        1 &amp;amp; z &amp;amp; z^2 \\
        \end{matrix}
$$
&lt;/div&gt;

&lt;h3 id=&quot;section-14&quot;&gt;加括号&lt;/h3&gt;
&lt;p&gt;如果要对矩阵加括号，可以像上文中提到的一样，使用\left与\right配合表示括号符号。也可以使用特殊的matrix。即替换&lt;code&gt;\begin{matrix}...\end{matrix}&lt;/code&gt;中的matrix为pmatrix，bmatrix，Bmatrix，vmatrix,Vmatrix.&lt;/p&gt;

&lt;div&gt;
如pmatrix:$\begin{pmatrix}1&amp;amp;2\\3&amp;amp;4\\ \end{pmatrix}$ bmatrix:$\begin{bmatrix}1&amp;amp;2\\3&amp;amp;4\\ \end{bmatrix}$ Bmatrix:$\begin{Bmatrix}1&amp;amp;2\\3&amp;amp;4\\ \end{Bmatrix}$ vmatrix:$\begin{vmatrix}1&amp;amp;2\\3&amp;amp;4\\ \end{vmatrix}$ Vmatrix:$\begin{Vmatrix}1&amp;amp;2\\3&amp;amp;4\\ \end{Vmatrix}$ 
&lt;/div&gt;

&lt;h3 id=&quot;section-15&quot;&gt;省略元素&lt;/h3&gt;
&lt;p&gt;可以使用\cdots $\cdots$ \ddots $\ddots$ \vdots $\vdots$来省略矩阵中的元素，如：&lt;/p&gt;

&lt;div&gt;
\begin{pmatrix}
     1 &amp;amp; a_1 &amp;amp; a_1^2 &amp;amp; \cdots &amp;amp; a_1^n \\
     1 &amp;amp; a_2 &amp;amp; a_2^2 &amp;amp; \cdots &amp;amp; a_2^n \\
     \vdots  &amp;amp; \vdots&amp;amp; \vdots &amp;amp; \ddots &amp;amp; \vdots \\
     1 &amp;amp; a_m &amp;amp; a_m^2 &amp;amp; \cdots &amp;amp; a_m^n    
\end{pmatrix}
&lt;/div&gt;

&lt;h3 id=&quot;section-16&quot;&gt;增广矩阵&lt;/h3&gt;
&lt;p&gt;增广矩阵需要使用前面的array来实现，如
&lt;code&gt;
$$ \left[
      \begin{array}{cc|c}
        1&amp;amp;2&amp;amp;3\\
        4&amp;amp;5&amp;amp;6
      \end{array}
    \right]
$$
&lt;/code&gt;
结果：&lt;/p&gt;

&lt;div&gt;
$$ \left[
      \begin{array}{cc|c}
        1&amp;amp;2&amp;amp;3\\
        4&amp;amp;5&amp;amp;6
      \end{array}
    \right]
$$
&lt;/div&gt;

&lt;h2 id=&quot;section-17&quot;&gt;对齐的公式&lt;/h2&gt;
&lt;p&gt;有时候可能需要一系列的公式中等号对齐，如：&lt;/p&gt;

&lt;div&gt;
$$\begin{align}
\sqrt{37} &amp;amp; = \sqrt{\frac{73^2-1}{12^2}} \\
 &amp;amp; = \sqrt{\frac{73^2}{12^2}\cdot\frac{73^2-1}{73^2}} \\ 
 &amp;amp; = \sqrt{\frac{73^2}{12^2}}\sqrt{\frac{73^2-1}{73^2}} \\
 &amp;amp; = \frac{73}{12}\sqrt{1 - \frac{1}{73^2}} \\ 
 &amp;amp; \approx \frac{73}{12}\left(1 - \frac{1}{2\cdot73^2}\right)
\end{align}$$
&lt;/div&gt;

&lt;p&gt;这需要使用形如&lt;code&gt;\begin{align}…\end{align}&lt;/code&gt;的格式，其中需要使用&amp;amp;来指示需要对齐的位置。请使用右键查看上述公式的代码。&lt;/p&gt;

&lt;h2 id=&quot;section-18&quot;&gt;分类表达式&lt;/h2&gt;
&lt;p&gt;定义函数的时候经常需要分情况给出表达式，可使用&lt;code&gt;\begin{cases}…\end{cases}&lt;/code&gt;。其中，使用\来分类，使用&amp;amp;指示需要对齐的位置。如：&lt;/p&gt;

&lt;div&gt;
$$
 f(n) =
\begin{cases}
n/2,  &amp;amp; \text{if $n$ is even} \\
3n+1, &amp;amp; \text{if $n$ is odd}  \\
\end{cases}
$$
&lt;/div&gt;

&lt;p&gt;上述公式的括号也可以移动到右侧，不过需要使用array来实现，如下：&lt;/p&gt;

&lt;div&gt;
$$\left.
\begin{array}{l}
\text{if $n$ is even:}&amp;amp;n/2\\
\text{if $n$ is odd:}&amp;amp;3n+1
\end{array}
\right\}
=f(n)$$
&lt;/div&gt;

&lt;p&gt;最后，如果想分类之间的垂直间隔变大，可以使用\[2ex]代替\来分隔不同的情况。(3ex,4ex也可以用，1ex相当于原始距离）。&lt;/p&gt;

&lt;h2 id=&quot;section-19&quot;&gt;数学符号查询&lt;/h2&gt;
&lt;p&gt;一般而言，从一个巨大的符号表中查询所需要的特定符号是一件令人沮丧的事情。在此向大家介绍一个$\LaTeX$手写符号识别系统，如下图：&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/ScK3R.png&quot; alt=&quot;Detexify 介绍 &quot; /&gt;&lt;/p&gt;

&lt;p&gt;尽情享用吧~ &lt;a href=&quot;http://detexify.kirelabs.org/classify.html&quot;&gt;Detexify²&lt;/a&gt;。&lt;/p&gt;

&lt;h2 id=&quot;section-20&quot;&gt;空间问题&lt;/h2&gt;
&lt;p&gt;在使用Latex公式时，有一些不会影响公式正确性，但却会使其看上去很槽糕的问题。&lt;/p&gt;

&lt;h3 id=&quot;frac&quot;&gt;不要在再指数或者积分中使用 \frac&lt;/h3&gt;
&lt;p&gt;在指数或者积分表达式中使用\frac会使表达式看起来不清晰，因此在专业的数学排版中很少被使用。应该使用一个水平的/来代替，效果如下：&lt;/p&gt;

&lt;div&gt;
$$
\begin{array}{cc}
\mathrm{Bad} &amp;amp; \mathrm{Better} \\
\hline \\
e^{i\frac{\pi}2} \quad e^{\frac{i\pi}2}&amp;amp; e^{i\pi/2} \\
\int_{-\frac\pi2}^\frac\pi2 \sin x\,dx &amp;amp; \int_{-\pi/2}^{\pi/2}\sin x\,dx \\
\end{array}
$$
&lt;/div&gt;

&lt;h3 id=&quot;mid---&quot;&gt;使用 \mid 代替 | 作为分隔符&lt;/h3&gt;
&lt;p&gt;符号|作为分隔符时有排版空间大小的问题，应该使用\mid代替。效果如下：&lt;/p&gt;

&lt;div&gt;
$$
\begin{array}{cc}
\mathrm{Bad} &amp;amp; \mathrm{Better} \\
\hline \\
\{x|x^2\in\Bbb Z\} &amp;amp; \{x\mid x^2\in\Bbb Z\} \\
\end{array}
$$
&lt;/div&gt;

&lt;h3 id=&quot;section-21&quot;&gt;多重积分&lt;/h3&gt;
&lt;p&gt;对于多重积分，不要使用\int\int此类的表达，应该使用\iint \iiint等特殊形式。效果如下：&lt;/p&gt;

&lt;div&gt;
$$
\begin{array}{cc}
\mathrm{Bad} &amp;amp; \mathrm{Better} \\
\hline \\
\int\int_S f(x)\,dy\,dx &amp;amp; \iint_S f(x)\,dy\,dx \\
\int\int\int_V f(x)\,dz\,dy\,dx &amp;amp; \iiint_V f(x)\,dz\,dy\,dx
\end{array}
$$
&lt;/div&gt;

&lt;p&gt;此外，在微分前应该使用\,来增加些许空间，否则$\TeX$会将微分紧凑地排列在一起。如下：&lt;/p&gt;

&lt;div&gt;
$$
\begin{array}{cc}
\mathrm{Bad} &amp;amp; \mathrm{Better} \\
\hline \\
\iiint_V f(x)dz dy dx &amp;amp; \iiint_V f(x)\,dz\,dy\,dx
\end{array}
$$
&lt;/div&gt;

&lt;h2 id=&quot;section-22&quot;&gt;连分数&lt;/h2&gt;
&lt;p&gt;书写连分数表达式时，请使用\cfrac代替\frac或者\over两者效果对比如下：&lt;/p&gt;

&lt;div&gt;
$$
x = a_0 + \cfrac{1^2}{a_1
          + \cfrac{2^2}{a_2
          + \cfrac{3^2}{a_3 + \cfrac{4^4}{a_4 + \cdots}}}} \tag{\cfrac}
$$
$$
x = a_0 + \frac{1^2}{a_1
          + \frac{2^2}{a_2
          + \frac{3^2}{a_3 + \frac{4^4}{a_4 + \cdots}}}} \tag{\frac}
$$
&lt;/div&gt;

&lt;h2 id=&quot;section-23&quot;&gt;方程组&lt;/h2&gt;
&lt;p&gt;使用&lt;code&gt;\begin{array} ... \end{array}&lt;/code&gt;与&lt;code&gt;\left\{…\right.&lt;/code&gt;配合，表示方程组，如：&lt;/p&gt;

&lt;div&gt;
$$
\left\{ 
\begin{array}{c}
a_1x+b_1y+c_1z=d_1 \\ 
a_2x+b_2y+c_2z=d_2 \\ 
a_3x+b_3y+c_3z=d_3
\end{array}
\right. 
$$
&lt;/div&gt;

&lt;p&gt;同时，还可以使用&lt;code&gt;\begin{cases}…\end{cases}&lt;/code&gt;表达同样的方程组，如：&lt;/p&gt;

&lt;div&gt;
$$\begin{cases}
a_1x+b_1y+c_1z=d_1 \\ 
a_2x+b_2y+c_2z=d_2 \\ 
a_3x+b_3y+c_3z=d_3
\end{cases}
$$
&lt;/div&gt;

&lt;p&gt;对齐方程组中的 = 号，可以使用 &lt;code&gt;\being{aligned} .. \end{aligned}&lt;/code&gt;，如：&lt;/p&gt;

&lt;div&gt;
$$
\left\{
\begin{aligned} 
a_1x+b_1y+c_1z &amp;amp;=d_1+e_1 \\ 
a_2x+b_2y&amp;amp;=d_2 \\ 
a_3x+b_3y+c_3z &amp;amp;=d_3 
\end{aligned} 
\right. 
$$
&lt;/div&gt;

&lt;p&gt;如果要对齐 = 号 和项，可以使用&lt;code&gt;\being{array}{列样式} .. \end{array}&lt;/code&gt;，如：&lt;/p&gt;

&lt;div&gt;
$$
\left\{
\begin{array}{ll}
a_1x+b_1y+c_1z &amp;amp;=d_1+e_1 \\ 
a_2x+b_2y &amp;amp;=d_2 \\ 
a_3x+b_3y+c_3z &amp;amp;=d_3 
\end{array} 
\right.
$$
&lt;/div&gt;

&lt;h2 id=&quot;section-24&quot;&gt;颜色&lt;/h2&gt;
&lt;p&gt;命名颜色是浏览器相关的，如果浏览器没有定义相关的颜色名称，则相关文本将被渲染为黑色。以下颜色是HTML4与CSS2标准中定义的一些颜色，其应该被大多数浏览器定义了。&lt;/p&gt;

&lt;div&gt;
$$
\begin{array}{|rc|}
\hline
\verb+\color{black}{text}+ &amp;amp; \color{black}{text} \\
\verb+\color{gray}{text}+ &amp;amp; \color{gray}{text} \\
\verb+\color{silver}{text}+ &amp;amp; \color{silver}{text} \\
\verb+\color{white}{text}+ &amp;amp; \color{white}{text} \\
\hline
\verb+\color{maroon}{text}+ &amp;amp; \color{maroon}{text} \\
\verb+\color{red}{text}+ &amp;amp; \color{red}{text} \\
\verb+\color{yellow}{text}+ &amp;amp; \color{yellow}{text} \\
\verb+\color{lime}{text}+ &amp;amp; \color{lime}{text} \\
\verb+\color{olive}{text}+ &amp;amp; \color{olive}{text} \\
\verb+\color{green}{text}+ &amp;amp; \color{green}{text} \\
\verb+\color{teal}{text}+ &amp;amp; \color{teal}{text} \\
\verb+\color{aqua}{text}+ &amp;amp; \color{aqua}{text} \\
\verb+\color{blue}{text}+ &amp;amp; \color{blue}{text} \\
\verb+\color{navy}{text}+ &amp;amp; \color{navy}{text} \\
\verb+\color{purple}{text}+ &amp;amp; \color{purple}{text} \\ 
\verb+\color{fuchsia}{text}+ &amp;amp; \color{magenta}{text} \\
\hline
\end{array}
$$
&lt;/div&gt;

&lt;p&gt;此外，HTML5与CSS3也定义了一些颜色名称，&lt;a href=&quot;http://www.w3.org/TR/css3-color/#svg-color&quot;&gt;参见&lt;/a&gt;。
同时，颜色也可以使用#rgb的形式来表示，r、g、b分别表示代表颜色值得16进制数，如：&lt;/p&gt;

&lt;div&gt;
$$
\begin{array}{|rrrrrrrr|}\hline
\verb+#000+ &amp;amp; \color{#000}{text} &amp;amp; &amp;amp; &amp;amp;
\verb+#00F+ &amp;amp; \color{#00F}{text} &amp;amp; &amp;amp; \\
&amp;amp; &amp;amp; \verb+#0F0+ &amp;amp; \color{#0F0}{text} &amp;amp;
&amp;amp; &amp;amp; \verb+#0FF+ &amp;amp; \color{#0FF}{text}\\
\verb+#F00+ &amp;amp; \color{#F00}{text} &amp;amp; &amp;amp; &amp;amp;
\verb+#F0F+ &amp;amp; \color{#F0F}{text} &amp;amp; &amp;amp; \\
&amp;amp; &amp;amp; \verb+#FF0+ &amp;amp; \color{#FF0}{text} &amp;amp;
&amp;amp; &amp;amp; \verb+#FFF+ &amp;amp; \color{#FFF}{text}\\
\hline
\end{array}
$$
&lt;/div&gt;

&lt;p&gt;&lt;a href=&quot;http://www.w3schools.com/html/html_colors.asp&quot;&gt;HTML色彩快速参考手册&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&quot;section-25&quot;&gt;公式标记与引用&lt;/h2&gt;
&lt;p&gt;使用\tag{yourtag}来标记公式，如果想在之后引用该公式，则还需要加上\label{yourlabel}在\tag之后，如：&lt;/p&gt;

&lt;div&gt;
$$
 a := x^2-y^3 \tag{*}\label{*}
$$
&lt;/div&gt;
&lt;p&gt;为了引用公式，可以使用&lt;code&gt;\eqref{rlabel}&lt;/code&gt;，如：&lt;/p&gt;
&lt;div&gt;
$$a+y^3 \stackrel{\eqref{*}}= x^2$$
&lt;/div&gt;

&lt;p&gt;可以看到，通过超链接可以跳转到被引用公式位置。&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Written with &lt;a href=&quot;https://stackedit.io/&quot;&gt;StackEdit&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

</description>
        <pubDate>Mon, 21 Mar 2016 00:29:22 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/21/mathjax-and-latex-syntax/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/21/mathjax-and-latex-syntax/</guid>
        
        <category>Latex</category>
        
        
        <category>latex</category>
        
      </item>
    
      <item>
        <title>深度学习卷积神经网络(斯坦福2016 CS231n)笔记</title>
        <description>&lt;h2 id=&quot;cs231n-lecture-3-&quot;&gt;CS231n Lecture 3 第三个视频笔记&lt;/h2&gt;

&lt;p&gt;SVM &amp;amp;&amp;amp; Softmax &amp;amp;&amp;amp; Learning Rates &amp;amp;&amp;amp; Gradient Check &amp;amp;&amp;amp; Mini-batch Gradient Descent&lt;/p&gt;

&lt;h3 id=&quot;svm-loss-function&quot;&gt;SVM loss function&lt;/h3&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;L_i_vector&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;W&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;scores&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;W&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;margins&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;maximum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scores&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;scores&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;margins&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;   &lt;span class=&quot;c&quot;&gt;#只是为了不等于1&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;loss_i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;margins&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
	&lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss_i&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Regularization motivation&lt;/p&gt;

&lt;p&gt;Take all X into account&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;k&quot;&gt;lambda&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;W&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;R(w)可能情况&lt;/p&gt;

&lt;p&gt;L2 regularization: R(W) = square(W)&lt;/p&gt;

&lt;p&gt;L1 regularization: R(W) = abs(W)       — feature selection&lt;/p&gt;

&lt;p&gt;Elastic net(L1+L2):R(W) = \beta.dot(W.dot(W)) + abs(W)&lt;/p&gt;

&lt;p&gt;Max norm regularization&lt;/p&gt;

&lt;p&gt;Dropout&lt;/p&gt;

&lt;h3 id=&quot;softmax-classifiermultinomial-logistic-regression&quot;&gt;Softmax Classifier(Multinomial Logistic Regression)&lt;/h3&gt;

&lt;p&gt;scores = unnormalized log probabilities of the classes.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;P(Y=k \| X=xi) = \frac{e^{S_k}}{\sum_j e^{S_j})} ;S = f(x_i;W,b)) = W x_i + b&lt;/script&gt;

&lt;p&gt;maximize the log likelihood or minimize the negative log likelihood&lt;/p&gt;

&lt;p&gt;Loss:
Li = -log(P(Y=yi|X=xi))&lt;/p&gt;

&lt;h4 id=&quot;softmax-vs-svm&quot;&gt;Softmax vs. SVM&lt;/h4&gt;

&lt;p&gt;if changing its score slightly,SVM doesn’t care.Softmax will change a lot.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;L = \frac{1}{N} \sum_i \sum_{j \neq y_i} \max(0,f_j - f_{y_i} + 1) + \lambda \sum_k \sum_l W_{k,l}^2&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;L = \underbrace { \frac{1}{N} \sum_i \sum_{j \neq y_i} \max(0,f_j - f_{y_i} + 1)}_{\text{data loss}} + \lambda \underbrace{\sum_k \sum_l W_{k,l}^2}_{\text{regularization loss}}&lt;/script&gt;

&lt;h2 id=&quot;cs231n-lecture-4--bp--nn&quot;&gt;CS231n Lecture 4 第四个视频 BP &amp;amp; NN&lt;/h2&gt;

&lt;p&gt;Chain rule&lt;/p&gt;

&lt;p&gt;calc $ f(x,y,z) = (x+y)z $ to understand activations of backprop.&lt;/p&gt;

&lt;p&gt;Another large example:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;f(w,x) = \frac{1}{1+e^{-(w_0 x_0 + w_1 x_1 + w_2)}}&lt;/script&gt;

&lt;p&gt;Jacobian Matrix
Diagonal Sparsity&lt;/p&gt;

&lt;h3 id=&quot;nn&quot;&gt;NN&lt;/h3&gt;

&lt;p&gt;超厉害的11行代码实现两层神经网络 &lt;a href=&quot;http://iamtrask.github.io/2015/07/12/basic-python-network/&quot;&gt;Basic Python Network&lt;/a&gt;&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;n&quot;&gt;X&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;array&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;array&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]])&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;syn0&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;	&lt;span class=&quot;c&quot;&gt;#Weight  Synapse 突触&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;syn1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;j&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;xrange&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;60000&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;l1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;/&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;exp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;syn0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))))&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;l2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;/&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;exp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;l1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;syn1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))))&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;l2_delta&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;l2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;l2&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;l2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;l1_delta&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;l2_delta&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;syn1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;l1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;l1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;syn1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;l1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;l2_delta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;syn0&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;l1_delta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Cell body : sum &amp;amp;&amp;amp; bias&lt;/p&gt;

&lt;h3 id=&quot;activation-funtion&quot;&gt;Activation funtion&lt;/h3&gt;

&lt;p&gt;sigmoid&lt;/p&gt;

&lt;p&gt;tanh(x):$ tanh(x) $&lt;/p&gt;

&lt;p&gt;ReLU:$max(0,x)$     make your classifier more faster, use as default&lt;/p&gt;

&lt;p&gt;Leaky ReLU: $max(0.1 x,x) $&lt;/p&gt;

&lt;p&gt;Maxout: $ max(w_1^T x +b_1,w_2^T x +b_2) $&lt;/p&gt;

&lt;p&gt;ELU:$ f(x) = \left{\begin{array}{c} 
x &lt;br /&gt;
\alpha (e^x-1)
\end{array}
\right.
$&lt;/p&gt;

&lt;h3 id=&quot;nn-1&quot;&gt;NN&lt;/h3&gt;

&lt;p&gt;数layers时，input layer不算&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html&quot;&gt;非常有意思的演示&lt;/a&gt;，将空间的转变演示出来了&lt;/p&gt;

&lt;p&gt;More always better~but you need avoid overfiting&lt;/p&gt;

&lt;p&gt;不同的layer不同的function，一般没人这么做&lt;/p&gt;

&lt;h2 id=&quot;lecture-5-nn-part2&quot;&gt;Lecture 5 NN part2&lt;/h2&gt;

&lt;p&gt;ConvNets 并不一定需要大量数据—— finetuning&lt;/p&gt;

&lt;p&gt;Don’t be overly ambitious.&lt;/p&gt;

&lt;h4 id=&quot;train-nn&quot;&gt;Train NN&lt;/h4&gt;
&lt;p&gt;Loop:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;Sample a batch of data&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Forward prop it through the graph, get loss&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Backprop to calculate the gradients&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Update the parameters using the gradient&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;training-neural-networks&quot;&gt;Training Neural Networks&lt;/h3&gt;

&lt;ol&gt;
  &lt;li&gt;One time setup&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;activation functions, preprocessing, weight
initialization, regularization, gradient checking&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Training dynamics&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;babysitting the learning process,
parameter updates, hyperparameter optimization&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Evaluation&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;model ensembles&lt;/p&gt;

&lt;h3 id=&quot;activation-functions&quot;&gt;Activation functions&lt;/h3&gt;

&lt;h4 id=&quot;sigmoid-funcition&quot;&gt;Sigmoid funcition&lt;/h4&gt;

&lt;p&gt;[0,1]&lt;/p&gt;

&lt;p&gt;Problem: downside
1. Saturated neurons “kill” the gradients&lt;/p&gt;

&lt;p&gt;x 在 -10 ,10之外的时候，好多节点就无法传递BP的gradients了&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Sigmoid outputs are not zero-centered&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;$ W_i $ Always all positive or all negative&lt;/p&gt;

&lt;p&gt;none-zero-centered function converge slower&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;exp() is a bit compute expensive&lt;/li&gt;
&lt;/ol&gt;

&lt;h4 id=&quot;tanhx&quot;&gt;Tanh(x)&lt;/h4&gt;

&lt;p&gt;[-1,1]
zero-centered&lt;/p&gt;

&lt;p&gt;Problem:
1. also Saturated&lt;/p&gt;

&lt;h3 id=&quot;relu&quot;&gt;ReLu&lt;/h3&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;f(x) = max(0,x)&lt;/script&gt;

&lt;p&gt;2012年的时候才解释==&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Don’t saturate (in +region)&lt;/li&gt;
  &lt;li&gt;Very computationally efficient&lt;/li&gt;
  &lt;li&gt;Converges much faster than sigmoid/tanh&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Problems:
1. not zero-centered output
2. An annoyance: When x&amp;lt;=0 , gradient is killed.问题是，初始化的时候初始化的不好，最后就会有几个Neuron从来没有Updated过。最后就是相当于Dead Neuron，数据从来没经历过这些节点。（常见解决方法（可能不管用）：初始化的时候with slightly positive bias eg:0.01）&lt;/p&gt;

&lt;h3 id=&quot;leaky-relu&quot;&gt;Leaky ReLU&lt;/h3&gt;

&lt;p&gt;为了解决ReLU里面的dead Neuron的问题&lt;/p&gt;

&lt;p&gt;&lt;script type=&quot;math/tex&quot;&gt;f(x) = max(0.01 x,x)&lt;/script&gt;
&lt;script type=&quot;math/tex&quot;&gt;f(x) = max(\alpha x,x)&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;Will not ‘die’&lt;/p&gt;

&lt;h3 id=&quot;elu-exponential-linear-units&quot;&gt;ELU exponential linear units&lt;/h3&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;% &lt;![CDATA[
f(x) = \begin{cases}
x,  &amp; \text{if $x &gt; 0$} \\
\alpha (e^x-1), &amp; \text{if $x \leq 0$}  \\
\end{cases} %]]&gt;&lt;/script&gt;

&lt;p&gt;2015年10月份的。。。。。[Clevert et al., 2015] 2页paper来证明。。&lt;/p&gt;

&lt;h3 id=&quot;maxout&quot;&gt;Maxout&lt;/h3&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;max(w_1^T x +b_1,w_2^T x +b_2)&lt;/script&gt;

&lt;p&gt;[Goodfellow et al., 2013]&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Does not have the basic form of dot product -&amp;gt; nonlinearity&lt;/li&gt;
  &lt;li&gt;Generalizes ReLU and Leaky ReLU&lt;/li&gt;
  &lt;li&gt;Linear Regime! Does not saturate! Does not die!&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Problem: doubles the number of parameters/neuron :(&lt;/p&gt;

&lt;h2 id=&quot;important&quot;&gt;Important!!!&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Use ReLU. Be careful with your learning rates&lt;/li&gt;
  &lt;li&gt;Try out Leaky ReLU / Maxout / ELU&lt;/li&gt;
  &lt;li&gt;Try out tanh but don’t expect much&lt;/li&gt;
  &lt;li&gt;Don’t use sigmoid&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;data-preprocessing&quot;&gt;Data Preprocessing&lt;/h2&gt;

&lt;p&gt;常用的就是Zero-center，一般不用normalize 或者 PCA或者whitening&lt;/p&gt;

&lt;p&gt;Not common to normalize variance, to do PCA or whitening&lt;/p&gt;

&lt;h3 id=&quot;in-practice-for-images-center-only&quot;&gt;In practice for Images: center only&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Subtract the mean image (e.g. AlexNet)&lt;/li&gt;
  &lt;li&gt;Subtract per-channel mean (e.g. VGGNet)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;weight-initialization&quot;&gt;Weight Initialization&lt;/h2&gt;

&lt;p&gt;Tiny number的话，最后W会很小&lt;/p&gt;

&lt;p&gt;如果超级混乱的init，disaster&lt;/p&gt;

&lt;h4 id=&quot;linear-activationsinit&quot;&gt;（linear activations）合理的Init&lt;/h4&gt;

&lt;p&gt;“Xavier initialization” [Glorot et al., 2010]&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;&lt;span class=&quot;nv&quot;&gt;W&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; np.random.randn&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;fan_in,fan_out&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt; / np.sqrt&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;fan_in&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# but when using the ReLU nonlinearity it breaks.&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Layer std decrease&lt;/p&gt;

&lt;h3 id=&quot;relu-1&quot;&gt;对于 ReLU&lt;/h3&gt;

&lt;p&gt;[He et al., 2015] note additional /2&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;&lt;span class=&quot;nv&quot;&gt;W&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; np.random.randn&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;fan_in,fan_out&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt; / np.sqrt&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;fan_in / 2&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id=&quot;section&quot;&gt;论文&lt;/h3&gt;

&lt;p&gt;Understanding the difficulty of training deep feedforward neural networks by Glorot and Bengio, 2010&lt;/p&gt;

&lt;p&gt;Exact solutions to the nonlinear dynamics of learning in deep linear neural networks by Saxe et al, 2013&lt;/p&gt;

&lt;p&gt;Random walk initialization for training very deep feedforward networks by Sussillo and Abbott, 2014&lt;/p&gt;

&lt;p&gt;Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification by He et al., 2015&lt;/p&gt;

&lt;p&gt;Data-dependent Initializations of Convolutional Neural Networks by Krähenbühl et al., 2015&lt;/p&gt;

&lt;p&gt;All you need is a good init, Mishkin and Matas, 2015&lt;/p&gt;

&lt;h3 id=&quot;batch-normalization&quot;&gt;Batch Normalization&lt;/h3&gt;

&lt;p&gt;[Ioffe and Szegedy, 2015]&lt;/p&gt;

&lt;p&gt;“you want unit gaussian activations? just make them so.”&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-text&quot; data-lang=&quot;text&quot;&gt;Usually inserted after Fully Connected(FC) / (or Convolutional, as we’ll see soon) layers, and before nonlinearity.&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
  &lt;li&gt;Improves gradient flow through the network&lt;/li&gt;
  &lt;li&gt;Allows higher learning rates&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Reduces the strong dependence on initialization&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;Acts as &lt;strong&gt;a form of regularization&lt;/strong&gt; in a funny way, and slightly reduces the need for dropout, maybe&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;可能有30%的速度衰减&lt;/p&gt;

&lt;p&gt;什么时候用BN？&lt;/p&gt;

&lt;h2 id=&quot;babysitting-the-learning-process&quot;&gt;Babysitting the Learning Process&lt;/h2&gt;

&lt;ol&gt;
  &lt;li&gt;Preprocess the data&lt;/li&gt;
  &lt;li&gt;Choose the architecture&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Make sure that you can overfit very small portion of the training data&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Learning rate = 1e-6,直接用眼看&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;cost: NaN almost always means high learning rate…&lt;/strong&gt;&lt;/p&gt;

&lt;h2 id=&quot;hyperparameter-optimization&quot;&gt;Hyperparameter Optimization&lt;/h2&gt;

&lt;p&gt;change range to calc learning rate&lt;/p&gt;

&lt;h4 id=&quot;grid-search-vs-random-search&quot;&gt;Grid Search vs random Search&lt;/h4&gt;
&lt;p&gt;Always use random search&lt;/p&gt;

&lt;h2 id=&quot;loss-function&quot;&gt;Loss function&lt;/h2&gt;

&lt;p&gt;&lt;a href=&quot;http://lossfunctions.tumblr.com/&quot;&gt;http://lossfunctions.tumblr.com/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;big gap = overfitting =&amp;gt; increase regularization strength?&lt;/p&gt;

&lt;p&gt;no gap =&amp;gt; increase model capacity?&lt;/p&gt;

&lt;h4 id=&quot;track-the-ratio-of-weight-updates--weight-magnitudes&quot;&gt;Track the ratio of weight updates / weight magnitudes:&lt;/h4&gt;

&lt;p&gt;ratio between the values and updates: ~ 0.0002 / 0.02 = 0.01 (about okay) want this to be somewhere around 0.001 or so&lt;/p&gt;

&lt;h2 id=&quot;important-1&quot;&gt;Important！！&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Regularization (Dropout etc)&lt;/li&gt;
  &lt;li&gt;Evaluation (Ensembles etc)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;lecture-6-nn&quot;&gt;Lecture 6 NN&lt;/h2&gt;

&lt;p&gt;full saturated or all zeros&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Parameter update schemes&lt;/li&gt;
  &lt;li&gt;Learning rate schedules&lt;/li&gt;
  &lt;li&gt;Dropout&lt;/li&gt;
  &lt;li&gt;Gradient checking&lt;/li&gt;
  &lt;li&gt;Model ensembles&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;parameter-update&quot;&gt;Parameter update&lt;/h3&gt;

&lt;p&gt;SGD is so slow&lt;/p&gt;

&lt;p&gt;SGD is a very slow progress along flat direction, jitter along steep one&lt;/p&gt;

&lt;h4 id=&quot;momentum-update&quot;&gt;Momentum update&lt;/h4&gt;

&lt;p&gt;比SGD快&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;n&quot;&gt;v&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mu&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;learning_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dx&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;X&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;解释&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Physical interpretation as ball rolling down the loss function + friction (mu coefficient).&lt;/li&gt;
  &lt;li&gt;mu = usually ~0.5, 0.9, or 0.99 (Sometimes a&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;优势&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Allows a velocity to “build up” along shallow directions&lt;/li&gt;
  &lt;li&gt;Velocity becomes damped in steep direction due to quickly changing sign&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;nesterov-momentum-update-nag&quot;&gt;Nesterov Momentum update (nag)&lt;/h4&gt;

&lt;p&gt;“lookahead” gradient step (bit different than original)&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;v_t = u v_{t-1} - \epsilon \nabla f(\phi_{t-1})
\phi_t = \phi_{t-1} - \mu v_{t-1} + ( 1 + \mu ) v_t&lt;/script&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;n&quot;&gt;v_prev&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;v&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mu&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;learning_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dx&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;X&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mu&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v_prev&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mu&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;** Common commercial network!!!!!!!**&lt;/p&gt;

&lt;h4 id=&quot;adagrad-update&quot;&gt;AdaGrad update&lt;/h4&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;n&quot;&gt;cache&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dx&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;**&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;learning_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dx&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sqrt&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cache&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;1e-7&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Added element-wise scaling of the gradient based on the historical  sum of squares in each dimension&lt;/p&gt;

&lt;p&gt;相当于自动变化Learning Rate&lt;/p&gt;

&lt;p&gt;现实中并不是这么好，到最后可能导致lr更大&lt;/p&gt;

&lt;h4 id=&quot;rmsprop-update&quot;&gt;RMSProp update&lt;/h4&gt;

&lt;p&gt;[Tieleman and Hinton, 2012]&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-python&quot; data-lang=&quot;python&quot;&gt;&lt;span class=&quot;n&quot;&gt;cache&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;decay_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cache&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;decay_rate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dx&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;**&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;learning_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dx&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;np&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sqrt&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cache&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;1e-7&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4 id=&quot;adam-update&quot;&gt;Adam update&lt;/h4&gt;

&lt;p&gt;合并了RMSProp和Momentum&lt;/p&gt;

&lt;p&gt;[Kingma and Ba, 2014]&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-bash&quot; data-lang=&quot;bash&quot;&gt;m,v &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;#... initialize caches to zeros&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; t in xrange&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;0,big_number&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;:
	&lt;span class=&quot;nv&quot;&gt;dx&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# evalute gradient&lt;/span&gt;
	&lt;span class=&quot;nv&quot;&gt;m&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; beta1*m + &lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;1-beta1&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;*dx
	&lt;span class=&quot;nv&quot;&gt;v&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; beta2*v + &lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;1-beta2&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;*&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;dx**2&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;
	m /&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;1&lt;/span&gt; - beta1**t
	v /&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;m&quot;&gt;1&lt;/span&gt; - beta2**t
	x +&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; - learning_rate * m / &lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;np.sqrt&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;v&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt; + 1e-7&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;用这个比较多&lt;/p&gt;

&lt;h3 id=&quot;learning-rate-as-a-hyperparameter&quot;&gt;learning rate as a hyperparameter&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Step decay&lt;/li&gt;
  &lt;li&gt;exponential decay&lt;/li&gt;
  &lt;li&gt;1/t decay&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;second-order-optimization-methods&quot;&gt;Second order optimization methods&lt;/h2&gt;

&lt;p&gt;使用了 Hessian，Invert the Hessian($O(n^3)$)&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;BGFS  把inverse Hessian with rank 1  优化成 $O(n^2)$&lt;/li&gt;
  &lt;li&gt;L-BGFS  Does not form/store the full inverse Hessian.&lt;/li&gt;
&lt;/ul&gt;

&lt;h5 id=&quot;l-bgfs&quot;&gt;L-BGFS&lt;/h5&gt;

&lt;p&gt;very heavy function.
In practice,we don’t use it.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Usually works very well in full batch, deterministic mode.  i.e. if you have a single, deterministic f(x) then L-BFGS will probably work very nicely&lt;/li&gt;
  &lt;li&gt;Does not transfer very well to mini-batch setting. Gives bad results. Adapting L-BFGS to large-scale, stochastic setting is an active area of research.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;in-practice&quot;&gt;In practice&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Adam is a good default choice in most cases&lt;/li&gt;
  &lt;li&gt;If you can afford to do full batch updates then try out L-BFGS (and don’t forget to disable all sources of noise) 一般不用&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;evaluation&quot;&gt;Evaluation&lt;/h2&gt;

&lt;p&gt;经验&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Train multiple independent models&lt;/li&gt;
  &lt;li&gt;At test time average their results&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Enjoy 2% extra performance&lt;/p&gt;

&lt;p&gt;??????????????&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;can also get a small boost from averaging multiple model checkpoints of a single model.&lt;/li&gt;
  &lt;li&gt;keep track of (and use at test time) a running average parameter vector:&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;regularizationdropout&quot;&gt;Regularization(Dropout)&lt;/h2&gt;

&lt;p&gt;&lt;img src=&quot;/images/20160321-dropout.jpg&quot; alt=&quot;dropout&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/20160321-why-use-dropout.jpg&quot; alt=&quot;why use dropout&quot; /&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Forces the network to have a redundant representation.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Dropout is training a large ensemble of models (that share parameters).&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Each binary mask is one model, gets trained on only ~one datapoint.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;in-practice-1&quot;&gt;In practice&lt;/h4&gt;

&lt;p&gt;drop的时候要注意&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Leave all input neurons turned on (no dropout).&lt;/li&gt;
  &lt;li&gt;With p=0.5, using all inputs in the forward pass would inflate the activations by 2x from what the network was “used to” during training! =&amp;gt; Have to compensate by scaling the activations back down by 1⁄2&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;???????&lt;/p&gt;

&lt;hr /&gt;
&lt;p&gt;ConvNets Face recognize [Taigman et al. 2014]&lt;/p&gt;

&lt;p&gt;Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition [Cadieu et al., 2014]&lt;/p&gt;

</description>
        <pubDate>Sun, 20 Mar 2016 20:55:20 +0000</pubDate>
        <link>http://next.sh/blog/2016/03/20/deep-learning-convolutional-neutral-networks-for-visual-recognition-winter-2016---cs231n-notes/</link>
        <guid isPermaLink="true">http://next.sh/blog/2016/03/20/deep-learning-convolutional-neutral-networks-for-visual-recognition-winter-2016---cs231n-notes/</guid>
        
        <category>Machine Learning</category>
        
        <category>Deep Learning</category>
        
        <category>CNN</category>
        
        
        <category>deep</category>
        
        <category>learning</category>
        
      </item>
    
  </channel>
</rss>