|
5 | 5 | <link>https://graphdeeplearning.github.io/authors/chaitanya-joshi/</link> |
6 | 6 | <atom:link href="https://graphdeeplearning.github.io/authors/chaitanya-joshi/index.xml" rel="self" type="application/rss+xml" /> |
7 | 7 | <description>NTU Graph Deep Learning Lab</description> |
8 | | - <generator>Source Themes Academic (https://sourcethemes.com/academic/)</generator><language>en-us</language><copyright>Xavier Bresson © 2020</copyright><lastBuildDate>Wed, 12 Feb 2020 16:08:39 +0800</lastBuildDate> |
| 8 | + <generator>Source Themes Academic (https://sourcethemes.com/academic/)</generator><language>en-us</language><copyright>Xavier Bresson © 2020</copyright><lastBuildDate>Tue, 03 Mar 2020 22:20:35 +0800</lastBuildDate> |
9 | 9 | <image> |
10 | 10 | <url>https://graphdeeplearning.github.io/images/icon_hu027d87ac1e37f4f802995042c9999554_21044_512x512_fill_lanczos_center_2.png</url> |
11 | 11 | <title>NTU Graph Deep Learning Lab</title> |
12 | 12 | <link>https://graphdeeplearning.github.io/authors/chaitanya-joshi/</link> |
13 | 13 | </image> |
14 | 14 |
|
| 15 | + <item> |
| 16 | + <title>Benchmarking Graph Neural Networks</title> |
| 17 | + <link>https://graphdeeplearning.github.io/project/benchmark/</link> |
| 18 | + <pubDate>Tue, 03 Mar 2020 22:20:35 +0800</pubDate> |
| 19 | + <guid>https://graphdeeplearning.github.io/project/benchmark/</guid> |
| 20 | + <description></description> |
| 21 | + </item> |
| 22 | + |
| 23 | + <item> |
| 24 | + <title>Benchmarking Graph Neural Networks</title> |
| 25 | + <link>https://graphdeeplearning.github.io/publication/dwivedi-2020-benchmark/</link> |
| 26 | + <pubDate>Mon, 02 Mar 2020 00:00:00 +0000</pubDate> |
| 27 | + <guid>https://graphdeeplearning.github.io/publication/dwivedi-2020-benchmark/</guid> |
| 28 | + <description></description> |
| 29 | + </item> |
| 30 | + |
15 | 31 | <item> |
16 | 32 | <title>Transformers are Graph Neural Networks</title> |
17 | 33 | <link>https://graphdeeplearning.github.io/post/transformers-are-gnns/</link> |
18 | 34 | <pubDate>Wed, 12 Feb 2020 16:08:39 +0800</pubDate> |
19 | 35 | <guid>https://graphdeeplearning.github.io/post/transformers-are-gnns/</guid> |
20 | 36 | <description><p>Engineer friends often ask me: Graph Deep Learning sounds great, but are there any big commercial success stories? Is it being deployed in practical applications?</p> |
21 | 37 | <p>Besides the obvious ones&ndash;recommendation systems at <a href="https://medium.com/pinterest-engineering/pinsage-a-new-graph-convolutional-neural-network-for-web-scale-recommender-systems-88795a107f48">Pinterest</a>, <a href="https://arxiv.org/abs/1902.08730">Alibaba</a> and <a href="https://blog.twitter.com/en_us/topics/company/2019/Twitter-acquires-Fabula-AI.html">Twitter</a>&ndash;a slightly nuanced success story is the <a href="https://arxiv.org/abs/1706.03762"><strong>Transformer architecture</strong></a>, which has <a href="https://openai.com/blog/better-language-models/">taken</a> <a href="https://www.blog.google/products/search/search-language-understanding-bert/">the</a> <a href="https://www.microsoft.com/en-us/research/project/large-scale-pretraining-for-response-generation/">NLP</a> <a href="https://ai.facebook.com/blog/roberta-an-optimized-method-for-pretraining-self-supervised-nlp-systems/">industry</a> <a href="https://blog.einstein.ai/introducing-a-conditional-transformer-language-model-for-controllable-generation/">by</a> <a href="https://nv-adlr.github.io/MegatronLM">storm</a>.</p> |
22 | | -<p>Through this post, I want to establish links between <a href="(https://graphdeeplearning.github.io/project/spatial-convnets/)">Graph Neural Networks (GNNs)</a> and Transformers. |
| 38 | +<p>Through this post, I want to establish links between <a href="https://graphdeeplearning.github.io/project/spatial-convnets/">Graph Neural Networks (GNNs)</a> and Transformers. |
23 | 39 | I&rsquo;ll talk about the intuitions behind model architectures in the NLP and GNN communities, make connections using equations and figures, and discuss how we could work together to drive progress.</p> |
24 | 40 | <p>Let&rsquo;s start by talking about the purpose of model architectures&ndash;<em>representation learning</em>.</p> |
25 | 41 | <hr> |
@@ -133,7 +149,7 @@ h_i^{\ell+1} = \text{LN} \left( \text{MLP} \left( \text{LN} \left( h_i^{\ell+1} |
133 | 149 | $$</p> |
134 | 150 | <blockquote> |
135 | 151 | <p>To be honest, I&rsquo;m not sure what the exact intuition behind the over-parameterized feed-forward sub-layer was and nobody seems to be asking questions about it, too! I suppose LayerNorm and scaled dot-products didn&rsquo;t completely solve the issues highlighted, so the big MLP is a sort of hack to re-scale the feature vectors independently of each other.</p> |
136 | | -<p ><a href= "mailto:chaitanya -[email protected]">Email me </a > if you know more! </p > |
| 152 | +<p ><a href= "mailto:chaitanya .[email protected]">Email me </a > if you know more! </p > |
137 | 153 | </blockquote> |
138 | 154 | <hr> |
139 | 155 | <p>The final picture of a Transformer layer looks like this:</p> |
@@ -226,9 +242,11 @@ In the example, $\mathcal{N}$(😆) $=$ { 😘, 😎, 😜, 🤩 }.</p> |
226 | 242 |
|
227 | 243 | </figure> |
228 | 244 |
|
229 | | -<blockquote> |
230 | | -<p>If we were to do multiple parallel heads of neighbourhood aggregation and replace summation over the neighbours $j$ with the attention mechanism, <em>i.e.</em>, a weighted sum, we&rsquo;d get the <b>Graph Attention Network</b> (GAT). Add normalization and the feed-forward MLP, and voila, we have a <b>Graph Transformer</b>!</p> |
231 | | -</blockquote> |
| 245 | +<div class="alert alert-note"> |
| 246 | + <div> |
| 247 | + If we were to do multiple parallel heads of neighbourhood aggregation and replace summation over the neighbours $j$ with the attention mechanism, <em>i.e.</em>, a weighted sum, we&rsquo;d get the <b>Graph Attention Network</b> (GAT). Add normalization and the feed-forward MLP, and voila, we have a <b>Graph Transformer</b>! |
| 248 | + </div> |
| 249 | +</div> |
232 | 250 | <hr> |
233 | 251 | <h3 id="sentences-are-fully-connected-word-graphs">Sentences are fully-connected word graphs</h3> |
234 | 252 | <p>To make the connection more explicit, consider a sentence as a fully-connected graph, where each word is connected to every other word. |
@@ -399,6 +417,11 @@ Similarly, DeepMind&rsquo;s <a href="https://arxiv.org/abs/1806.01261 |
399 | 417 | For a code walkthrough, the DGL team has <a href="https://docs.dgl.ai/en/latest/tutorials/models/4_old_wines/7_transformer.html">a nice tutorial</a> on seq2seq as a graph problem and building Transformers as GNNs.</p> |
400 | 418 | <p><strong>In our next post, we&rsquo;ll be doing the reverse: using GNN architectures as Transformers for NLP (based on the Transformers library by <a href="https://github.com/huggingface/transformers">🤗 HuggingFace</a>).</strong></p> |
401 | 419 | <p>Finally, we wrote <a href="https://graphdeeplearning.github.io/publication/xu-2019-multi/">a recent paper</a> applying Transformers to sketch graphs. Do check it out!</p> |
| 420 | +<hr> |
| 421 | +<h4 id="updates">Updates</h4> |
| 422 | +<p>The post has also been translated to <a href="https://mp.weixin.qq.com/s/DABEcNf1hHahlZFMttiT2g">Chinese</a>. |
| 423 | +Do join the discussion on <a href="https://twitter.com/chaitjo/status/1233220586358181888?s=20">Twitter</a> or <a href="https://www.reddit.com/r/MachineLearning/comments/fb86mo/d_transformers_are_graph_neural_networks_blog/">Reddit</a>!</p> |
| 424 | +<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Transformers are a special case of Graph Neural Networks. This may be obvious to some, but the following blog post does a good job at explaining these important concepts. <a href="https://t.co/H8LT2F7LqC">https://t.co/H8LT2F7LqC</a></p>&mdash; Oriol Vinyals (@OriolVinyalsML) <a href="https://twitter.com/OriolVinyalsML/status/1233783593626951681?ref_src=twsrc%5Etfw">February 29, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> |
402 | 425 | </description> |
403 | 426 | </item> |
404 | 427 |
|
@@ -491,10 +514,27 @@ Examples include chemical graphs, computer graphics, social networks, genetics, |
491 | 514 | <p>Graph-structured data can be large and complex (in the case of social networks, on the scale of billions), and is a natural target for machine learning applications. |
492 | 515 | However, designing models for learning from non-Euclidean data is challenging as there are no familiar properties such as coordinate systems, vector space structure, or shift invariance.</p> |
493 | 516 | <p><strong>Graph/Geometric Deep Learning</strong> is an umbrella term for emerging techniques attempting to generalize deep neural networks to non-Euclidean domains such as graphs and manifolds [<a href="https://arxiv.org/abs/1611.08097">Bronstein <em>et al.</em>, 2017</a>]. |
494 | | -We are interested to designing neural networks for graphs of arbitrary topologies and structures in order to solve generic graph problems, such as vertex classification, graph classification, graph regression, and graph generation. |
495 | | -These Graph Neural Network (GNN) architectures are used as backbones for challenging domain-specific applications in chemistry, social networks or graphics.</p> |
496 | | -<p><img src="gnn-layer.png" alt="GNN Layer"> |
497 | | -<em>GNNs iteratively build representations of graphs through recursive neighborhood aggregation (or message passing), where each graph node gathers features from its neighbors to represent local graph structure.</em></p> |
| 517 | +We are interested to designing neural networks for graphs of arbitrary topologies and structures in order to solve <em>generic</em> graph problems, such as vertex classification, graph classification, graph regression, and graph generation.</p> |
| 518 | +<p>These Graph Neural Network (GNN) architectures are used as backbones for challenging domain-specific applications in a myriad of domains, including chemistry, social networks, recommendations and computer graphics.</p> |
| 519 | +<hr> |
| 520 | +<h2 id="basic-formalism">Basic Formalism</h2> |
| 521 | +<p>Each GNN layer computes $d$-dimensional representations for the nodes/edges of the graph through recursive neighborhood diffusion (or message passing), where each graph node gathers features from its neighbors to represent local graph structure. |
| 522 | +Stacking $L$ GNN layers allows the network to build node representations from the <strong>$L$-hop neighborhood</strong> of each node.</p> |
| 523 | +<p><img src="gnn-layer.png" alt="GNN Layer"></p> |
| 524 | +<p>Let $h_i^{\ell}$ denote the feature vector at layer $\ell$ associated with node $i$. |
| 525 | +The updated features $h_i^{\ell+1}$ at the next layer $\ell+1$ are obtained by applying non-linear transformations to the central feature vector $h_i^{\ell}$ and the feature vectors $h_{j}^{\ell}$ for all nodes $j$ in the neighborhood of node $i$ (defined by the graph structure). |
| 526 | +This guarantees the transformation to build local reception fields, such as in standard ConvNets for computer vision, and be invariant to both graph size and vertex re-indexing.</p> |
| 527 | +<p>Thus, the most generic version of a feature vector $h_i^{\ell+1}$ at vertex $i$ at the next layer in the graph network is: |
| 528 | +\begin{equation} |
| 529 | +h_{i}^{\ell+1} = f \left( \ h_i^{\ell} \ , \ { h_{j}^{\ell}: j \rightarrow i } \ \right) , |
| 530 | +\end{equation} |
| 531 | +where ${ j \rightarrow i }$ denotes the set of neighboring nodes $j$ pointed to node $i$, which can be replaced by ${ j \in \mathcal{N}_i }$, the set of neighbors of node $i$, if the graph is undirected. In other words, a GNN is defined by a mapping $f$ taking as input a vector $h_i^{\ell}$ (the feature vector of the center vertex) as well as an un-ordered set of vectors ${ h_{j}^{\ell}}$ (the feature vectors of all neighboring vertices).</p> |
| 532 | +<p>The arbitrary choice of the mapping $f$ defines an instantiation of a class of GNNs, <em>e.g.</em>, <a href="https://arxiv.org/abs/1706.02216">GraphSage</a>, <a href="https://arxiv.org/abs/1611.08402">MoNet</a>, <a href="https://arxiv.org/abs/1710.10903">GAT</a>, etc. |
| 533 | +For an illustration, here&rsquo;s a simple-yet-effective Graph ConvNet from <a href="https://arxiv.org/abs/1605.07736">Sukhbaatar <em>et al.</em>, 2016</a>: |
| 534 | +\begin{equation} |
| 535 | +h_{i}^{\ell+1} = \text{ReLU} \Big( U^{\ell} h_{i}^{\ell} + \sum_{j \in \mathcal{N}_i} V^{\ell} h_{j}^{\ell} \Big), |
| 536 | +\end{equation} |
| 537 | +where $U^{\ell}, V^{\ell} \in \mathbb{R}^{d \times d}$ are the learnable parameters.</p> |
498 | 538 | </description> |
499 | 539 | </item> |
500 | 540 |
|
|
0 commit comments