Merge pull request swiftlang#167 from dabelknap/documentation

Add verbatim tokens to documentation

Author: allevato

Documentation/PrettyPrinter.md

@@ -25,8 +25,8 @@ tokens and end with *close* tokens. These tokens must always be paired.
 
 The different types of tokens are represented as a Token `enum` within the code.
 The available cases are: `syntax`, `break`, `open`, `close`, `newlines`,
-`comment`, and `reset`. The behavior of each of them is described below with
-pseudocode examples.
+`comment`, `reset`, and `verbatim`. The behavior of each of them is described
+below with pseudocode examples.
 
 See: [`Token.swift`](../Sources/SwiftFormatPrettyPrint/Token.swift)
 
@@ -239,6 +239,38 @@ Tokens = [block(" Block comment\n   Second Line ")]
 Tokens = [docBlock(" Doc Block comment\n  * Second line *")]
 ```
 
+#### Verbatim
+
+Verbatim tokens are used to print text verbatim without any formatting apart
+from applying a global indentation. They have a length set to the maximum line
+width. They are typically used to handle syntax types that are classed as
+"unknown" by SwiftSyntax. In these cases, we don't have access to the
+substructure of the syntax node a manner useful for formatting, so we print them
+verbatim. The indentation for verbatim tokens is applied to the first line of
+the text. The relative indentation of subsequent lines is preserved unless they
+have less indentation than the first line, in which case we set the indentation
+of those lines equal to the first.
+
+```
+// Consider "ifnt", an unknown syntax structure:
+
+if someCondition {
+    ifnt anotherCondition {
+      let a = 123
+  let b = 456
+    }
+}
+
+// The pretty-printer will transform this into:
+
+if someCondition {
+  ifnt anotherCondition {
+    let a = 123
+  let b = 456
+  }
+}
+```
+
 ### Token Generation
 
 Token generation begins with the abstract syntax tree (AST) of the Swift source