Restricted Boltzmann machine

Restricted Boltzmann machine

Diagram of a restricted Boltzmann machine with three visible units and four hidden units(no bias units).

A restricted Boltzmann machine(RBM)is a generative stochastic arti?cial neural network that can learn a probability distribution over its set of inputs. RBMs were initially invented under the name Harmo-nium by Paul Smolensky in1986,[1]but only rose to prominence after Geo?rey Hinton and collaborators invented fast learning algorithms for them in the mid-2000s.RBMs have found applications in dimensionality reduction,[2]classi?cation,[3]collaborative?ltering,[4] feature learning[5]and topic modelling.[6]They can be trained in either supervised or unsupervised ways, depending on the task.

As their name implies,RBMs are a variant of Boltzmann machines,with the restriction that their neurons must form a bipartite graph:a pair of nodes from each of the two groups of units,commonly referred to as the“visi-ble”and“hidden”units respectively,may have a symmet-ric connection between them,and there are no connec-tions between nodes within a group.By contrast,“unre-stricted”Boltzmann machines may have connections be-tween hidden units.This restriction allows for more e?-cient training algorithms than are available for the general class of Boltzmann machines,in particular the gradient-based contrastive divergence algorithm.[7]

Restricted Boltzmann machines can also be used in deep learning networks.In particular,deep belief networks can be formed by“stacking”RBMs and optionally?ne-

tuning the resulting deep network with gradient descent and backpropagation.[8]

1Structure

The standard type of RBM has binary-valued (Boolean/Bernoulli)hidden and visible units,and consists of a matrix of weights W=(w i,j)(size m×n) associated with the connection between hidden unit h j and visible unit v i,as well as bias weights(o?sets)a i for the visible units and b j for the hidden units.Given these, the energy of a con?guration(pair of boolean vectors) (v,h)is de?ned as

E(v,h)=?

∑

i

a i v i?

∑

j

b j h j?

∑

i

∑

j

v i w i,j h j or,in vector form,

E(v,h)=?a T v?b T h?v T W h

This energy function is analogous to that of a Hop?eld network.As in general Boltzmann machines,probability distributions over hidden and/or visible vectors are de-?ned in terms of the energy function:[9]

P(v,h)=

1

Z

e?E(v,h)

where Z is a partition function de?ned as the sum of e?E(v,h)over all possible con?gurations(in other words, just a normalizing constant to ensure the probability dis-tribution sums to1).Similarly,the(marginal)probability of a visible(input)vector of booleans is the sum over all possible hidden layer con?gurations:[9]

P(v)=

1

Z

∑

h

e?E(v,h)

Since the RBM has the shape of a bipartite graph,with no intra-layer connections,the hidden unit activations are mutually independent given the visible unit activations and conversely,the visible unit activations are mutually independent given the hidden unit activations.[7]That is, for m visible units and n hidden units,the conditional

1

24REFERENCES probability of a con?guration of the visible units v,given

a con?guration of the hidden units h,is

P(v|h)=

m

∏

i=1

P(v i|h)

Conversely,the conditional probability of h given v is

P(h|v)=

n

∏

j=1

P(h j|v)

The individual activation probabilities are given by

P(h j=1|v)=σ(b j+∑m

i=1

w i,j v i)and

P(v i=1|h)=σ(

a i+

∑n

j=1

w i,j h j

)

whereσdenotes the logistic sigmoid.

The visible units of RBM can be multinomial,although the hidden units are Bernoulli.In this case,the logis-tic function for visible units is replaced by the Softmax function

P(v k i=1|h)=

exp(a k i+Σj h j W k ij)

ΣK

k=1

exp(a k i+Σj h j W k ij)

where K is the number of discrete values that the visible values have.They are applied in Topic Modeling,[6]and RecSys.[4]

1.1Relation to other models

Restricted Boltzmann machines are a special case of Boltzmann machines and Markov random?elds.[10][11] Their graphical model corresponds to that of factor anal-ysis.[12]

2Training algorithm

Restricted Boltzmann machines are trained to maximize the product of probabilities assigned to some training set V(a matrix,each row of which is treated as a visible vec-tor v),

arg max

W ∏

v∈V

P(v)

or equivalently,to maximize the expected log probability of V:[10][11]

arg max

W E

[

∑

v∈V

log P(v)

]

The algorithm most often used to train RBMs,that is,

to optimize the weight vector W,is the contrastive di-

vergence(CD)algorithm due to Hinton,originally devel-

oped to train PoE(product of experts)models.[13][14]The

algorithm performs Gibbs sampling and is used inside

a gradient descent procedure(similar to the way back-

propagation is used inside such a procedure when training

feedforward neural nets)to compute weight update.

The basic,single-step contrastive divergence(CD-1)pro-

cedure for a single sample can be summarized as follows:

1.Take a training sample v,compute the probabilities

of the hidden units and sample a hidden activation

vector h from this probability distribution.

https://www.sodocs.net/doc/742686707.html,pute the outer product of v and h and call this

the positive gradient.

3.From h,sample a reconstruction v'of the visible

units,then resample the hidden activations h'from

this.(Gibbs sampling step)

https://www.sodocs.net/doc/742686707.html,pute the outer product of v'and h'and call this

the negative gradient.

5.Let the weight update to w i,j be the positive gradi-

ent minus the negative gradient,times some learning

rate:?w i,j=?(vh T?v′h′T).

The update rule for the biases a,b is de?ned analogously.

A Practical Guide to Training RBMs written by Hinton

can be found in his homepage.[9]

3See also

?Autoencoder

?Deep learning

?Helmholtz machine

?Hop?eld network

4References

[1]Smolensky,Paul(1986).“Chapter6:Information Pro-

cessing in Dynamical Systems:Foundations of Harmony

Theory”.In Rumelhart,David E.;McLelland,James L.

[[Connectionism|Parallel Distributed Processing]]:Explo-

rations in the Microstructure of Cognition,Volume1:Foun-

dations.MIT Press.pp.194–281.ISBN0-262-68053-

X.Wikilink embedded in URL title(help)

[2]Hinton,G. E.;Salakhutdinov,R.R.(2006).

“Reducing the Dimensionality of Data with Neu-

ral Networks”.Science313(5786):504–507.

doi:10.1126/science.1127647.PMID16873662.

3 [3]Larochelle,H.;Bengio,Y.(2008).“Proceedings of

the25th international conference on Machine learning-

ICML'08”.p.536.doi:10.1145/1390156.1390224.

ISBN9781605582054.|chapter=ignored(help)

[4]Salakhutdinov,R.;Mnih, A.;Hinton,G.(2007).

“Proceedings of the24th international confer-

ence on Machine learning-ICML'07”.p.791.

doi:10.1145/1273496.1273596.ISBN9781595937933.

|chapter=ignored(help)

[5]Coates,Adam;Lee,Honglak;Ng,Andrew Y.(2011).

“An analysis of single-layer networks in unsupervised fea-

ture learning”.International Conference on Arti?cial In-

telligence and Statistics(AISTATS).

[6]Ruslan Salakhutdinov and Geo?rey Hinton(2010).

Replicated softmax:an undirected topic model.Neural

Information Processing Systems23.

[7]Miguelá.Carreira-Perpi?án and Geo?rey Hinton(2005).

On contrastive divergence learning.Arti?cial Intelligence

and Statistics.

[8]Hinton,G.(2009).“Deep belief networks”.Scholarpedia

4(5):5947.doi:10.4249/scholarpedia.5947.

[9]Geo?rey Hinton(2010).A Practical Guide to Training Re-

stricted Boltzmann Machines.UTML TR2010–003,Uni-

versity of Toronto.

[10]Sutskever,Ilya;Tieleman,Tijmen(2010).“On the con-

vergence properties of contrastive divergence”.Proc.

13th Int'l Conf.on AI and Statistics(AISTATS).

[11]Asja Fischer and Christian Igel.Training Restricted

Boltzmann Machines:An Introduction.Pattern Recog-

nition47,pp.25-39,2014

[12]María Angélica Cueto;Jason Morton;Bernd Sturmfels

(2010).“Geometry of the restricted Boltzmann machine”.

Algebraic Methods in Statistics and Probability(American

Mathematical Society)516.arXiv:0908.4425.

[13]Geo?rey Hinton(1999).Products of Experts.ICANN

1999.

[14]Hinton,G. E.(2002).“Training Products

of Experts by Minimizing Contrastive Diver-

gence”.Neural Computation14(8):1771–1800.

doi:10.1162/089976602760128018.PMID12180402.

5External links

?Introduction to Restricted Boltzmann Machines.

Edwin Chen’s blog,July18,2011.

46TEXT AND IMAGE SOURCES,CONTRIBUTORS,AND LICENSES 6Text and image sources,contributors,and licenses

6.1Text

?Restricted Boltzmann machine Source:https://www.sodocs.net/doc/742686707.html,/wiki/Restricted%20Boltzmann%20machine?oldid=632894634Contrib-utors:Shd,Glenn,Qwertyus,Kri,Gareth Jones,Mcld,Hongooi,Dicklyon,Abhineetnazi,Tomtheebomb,Dsimic,Arsi Warrior,Yobot, UbaiSandouk,Racerx11,Leopd,Enerjiparki,Sevensharpnine,Velvel2and Anonymous:20

6.2Images

?File:Restricted_Boltzmann_machine.svg Source:https://www.sodocs.net/doc/742686707.html,/wikipedia/commons/e/e8/Restricted_Boltzmann_ machine.svg License:CC-BY-SA-3.0Contributors:Own work Original artist:Qwertyus

6.3Content license

?Creative Commons Attribution-Share Alike3.0

格子Boltzmann

格子Boltzmann 方法模拟C/C 复合材料 颗粒沉积过程 罗思璇 () Particle Deposition Process Simulation in C/C Composites by Lattice-Boltzmann Method Luo Sixuan () Abstract: Lattice Boltzmann method is used here to study the particle deposition process on C/C composites surface. This method considered the boudary condition change during particle deposition. Finally, the deposition pattern is obtained. Keywords: LB Method; flow-particle coupling; C/C composites; deposition 摘要：本文使用格子Boltzmann 方法研究了固体火箭发动机中C/C 复合材料表面上颗粒的沉积模态。该方法考虑了沉积过程中边界形貌的变化对流场的影响，最终得到了颗粒在碳纤维表面的沉积形态。 关键词：LB 方法；流固耦合；C/C 复合材料；沉积 0 引言 C/C 复合材料是目前新材料领域重点研究和开发的一种新型超高温热结构材料，具有密度小，比强度大、热膨胀系数低、热导率高等特点，是理想的航空航天高温材料[1, 2]。 C/C 复合材料在工作过程中其表面流过的工质为高温燃气。高温燃气中通常带有燃烧产生的固体颗粒，如选用较高比冲的含铝推进剂时会产生一定量的凝聚相(Al2O3颗粒)。固体颗粒在C/C 复合材料表面的沉积、冲刷及烧蚀会造成材料内型面的破坏，甚至影响气动性能。 本文使用格子Boltzmann 方法模拟C/C 复合材料中碳纤维上颗粒沉积过程及形态。 1模拟流场的格子Boltzmann 模型 格子Boltzmann 方法是近二十年来刚发展起来的，一种以“半晶格分离法”为处理方式的新型热量逐级传递数值方法,最初是在研究电磁场中的流动现象时被提出的，并且该方法可以确定流体域、固体域和温度场在边界处的连续性,十分适合针对复杂几何形状流固耦合传热问题的数值分析。与传统的经典CFD 方法相比，格子波尔兹曼算法具有很多优点。因而近年来受到国内外学者的广泛关注,并迅速在气固两相流和传热等研究领域得到应用。 格子Boltzmann 方法将流体抽象为微观的虚拟颗粒，通过这些颗粒在规则的网格点上进行碰撞和迁移来达到模拟流场的目的。分布函数f i (x ,t )表示t 时刻，x 网格点上，速度为c i 流体颗粒的概率密度，流场的宏观量通过对分布函数进行统计而得到。本文使用D3Q15模型模拟流场，流体宏观密度ρ和动量ρu 计算如下： 10 Q i i f ρ-==∑，1 Q i i i f ρ-==∑u c (1) 本文使用BGK 碰撞算子[3]，流场演化方程为： eq (,)(,)[(,)(,)]i i i i i f x t t t f x t f x t f x t τ+??+?-=-c (2) 其中?t 为时间步长，τ为无量纲松弛时间，eq i f 为平衡态分布函数，在D2Q9模型中如下计算：