[WIP] Reactive rerendering #282

wycats · 2015-02-03T03:15:09Z

This is a WIP PR to implement React-inspired fast rerendering of HTMLBars templates.

One important improvement we can make over the React model is eliminating any work for static portions of the template. Because HTMLBars templates are declarative, not imperative like React's model, the compiled template knows which parts of the DOM can change (because they were created through {{}}s) and which parts will never change.

Compiled HTMLBars templates already separate out the process of building the static "skeleton" from the process of filling it in, and this PR allows us to update the dynamic portions of the result of a previous render. Conceptually, this allows us to run the same "rendering" code multiple times, making re-renders fast without having to expensively diff parts of the DOM that will never change.

Because conditionals and loops are also declarative, we can eliminate the cognitive overhead React requires when conditionally rendering content. Instead of requiring that the user remember to insert null into a conditional slot, the template engine will know the position of an {{#if}} block, regardless of whether the conditional is truthy or not. There is no way for the user to screw this up and produce an off-by-one misalignment, which circumvents the diffing algorithm and triggers a full re-render.

TODO

wycats · 2015-02-03T03:15:33Z

The implementation in this PR is extremely WIP, so don't let the implementation scare you 😉

lin7sh · 2015-02-03T14:18:00Z

looks amazing!

cibernox · 2015-02-05T11:22:43Z

One question about how the diff-ing works.
If an template contains a triple moustache {{{}}} where I output trusted HTML and I re-render that template, the content of that moustache will be diffed?

That would be just beautiful.

mixonic · 2015-02-05T15:31:30Z

@cibernox Others can correct me if I am wrong, but this work does not make any effort toward that aim.

In fact, most libraries the perform dom-diffing are performing dom-like-data-tree diffing. The "virtual DOM" as we call it. They don't actually parse HTML into DOM, they require you to use a DSL to write dom-like objects that are later translated into DOM.

Since a value inside a {{{}}} is an HTML string, the best way to parse it into DOM is going to be using innerHTML. This returns real DOM objects, we any implementation wouldn't be diffing a virtual dom-like data structure but actual DOM. There might be performance gains, but it is definitely unlike any diffing solution out there as I understand them.

Additionally, the case of {{{}}} is extremely rare. Ember applications use templates heavily, and it is there we can do the work of determining structure upfront (much as virtual-dom DSLs do) and make an incredible impact.

Hope that makes some sense!

stefanpenner · 2015-02-05T15:45:29Z

Since a value inside a {{{}}} is an HTML string, the best way to parse it into DOM is going to be using innerHTML. This returns real DOM objects, we any implementation wouldn't be diffing a virtual dom-like data structure but actual DOM. There might be performance gains, but it is definitely unlike any diffing solution out there as I understand them.

or just old{{{}}} === new{{{}}} :P

cibernox · 2015-02-05T15:59:37Z

I imagined that.

I had a very particular use-case where some cards contain safe HTML coming directly from a CMS. Although I can imagine a workaround. Converting the HTML to templates in runtime (I would have to include the HTMLBars compiler) and render views dinamically instead of using {{{}}}.

Sounds possible in my head :P

rwjblue · 2015-02-06T01:26:28Z

packages/htmlbars-compiler/lib/compiler.js

@@ -1,6 +1,7 @@
 /*jshint evil:true*/
 import { preprocess } from "../htmlbars-syntax/parser";
 import TemplateCompiler from "./template-compiler";
+import { wrap } from "../htmlbars-runtime/hooks";


shouldn't be importing from htmlbars-runtime from within htmlbars-compiler for anything other than tests (the runtime is not available within Ember for example). If the wrap function needs to be shared and available to compiler (and ultimately to the consuming library) it should be in htmlbars-utils.

Nice. If utils is imported into both runtime and compiler, that's the right place for this for sure.

Previously, if you wanted to re-render an HTMLBars template, the template would create a brand new clone of the cached fragment and create a new set of morphs. This resulted in unnecessary work, since the static portions of the rendered DOM already exist, and do not need to be re-cloned and re-inserted into DOM. This change begins the process of allowing an HTMLBars template to update the output of a previous render, rather than starting from scratch. It introduces the concept of a (very WIP atm) “result” of a previous render, which is passed back in to subsequent renders. At the moment, the “result” is the fragment and morphs of the previous render. Soon, it will become a more opaque data structure that includes the results of sub-programs.

This commit introduces the notion of a “result” data type, which encapsulates all of the state needed to be passed to a compiled template in order to successfully re-render with an existing element. Rather than returning a document fragment, `render()` now returns this result type. For compatibility, consumers of this API will need to use the `fragment` property of the result rather than the result directly. In order to re-render, you can pass the last result as the `lastResult` option, and it will use that element and morph to update. One small additional change: contextual elements must now be passed in the options hash. We decided to make this change as the list of arguments was growing unwieldy, and we wanted to make this durable to changes in the future. We will also investigate incorporating block params into the options hash.

This change is an important step towards our efforts to offer rerendering of an existing element, rather than creating a new document fragment each time a template’s context changes. In the old compiled templates, the monolithic render method did all of the following: 1. Decide whether to use a cached fragment or build a new one from the build method. 2. Clone the fragment if possible / necessary 3. Create variables pointing at the fragment’s children (element0, etc.) 4. Create a bunch of pointers (morphs) with those element variables 5. Fill in the pointers (“populate”) Because all of the “pointers” necessary to update the DOM in the future were created as variables in the function scope, access to them was lost as soon as the function exited. This change is designed to make it possible to run step (4) above over and over again on the same piece of DOM. In order to make that possible, we changed the first few steps so that the pointers into the fragment are returned as an array, where they can be passed into a standalone “populate” function. For future readers, the “pointers” into the DOM are now called “render nodes”. These render nodes can point at either elements, attributes, or ranges. Expect these nodes to gain more responsibilities in the future. We also extracted the cached fragment logic into a hook, rather than repeating the completely static content in every template, significantly shrinking the size of compiled templates. The next step is to break up the `render` method into multiple pieces, so that the part that fills in the render nodes is completely standalone, and can be called by passing in the previous render nodes. (We wrote down our original design that we are still more-or-less iterating towards at https://gist.github.com/wycats/fc53be32abee6c5ca0f4.)

This commit finishes the process of breaking apart the template into discrete phases: 1. Building the static DOM “skeleton” 2. Creating morphs (render nodes) that point into the dynamic portions of the DOM skeleton 3. Filling in the dynamic portions with content from the context object By breaking these into discrete steps, we can extract the code that coordinates the steps into library code, significantly reducing the size of compiled templates. This also finally unlocks a very nice API for rerendering a template that reuses the previously generated DOM, rather than starting from a new “skeleton” each time. Not only is updating an existing DOM much less expensive, it allows us to rerender large portions of an application’s UI without losing state like cursor position or scroll position. This technique was pioneered by React. Calling `render()` on a template returns a new object that has both a `fragment` property on it as well as a `rerender()` method. To use re-rendering, call `render(ctx, env, options)` on a template, then save the result. You can put the fragment into the DOM, and when you want to update the template, simply call the `rerender()` method, passing the appropriate context. ```js var result = template.render(model, env, options); document.appendChild(result.fragment); // … later result.rerender(newModel); ```

Previously, invoking a block helper would always return a string or element that would be inserted into the parent fragment. This model assumed that the root-most template was always being rendered exactly once. With the introduction of re-rendering, block helpers need to handle this case as well. Specifically, instead of just creating rendering a new element each time, they should populate a provided morph with the contents they want to display. As an optimization, you can also save off the result of the previous render and `rerender()` it. This change is necessary to support re-rendering, but we consider it a low-level API. We intended to create a high-level API for authors to use that handles the most frequent rendering and re-rendering patterns. In particular, the test titled “Templates with block helpers - re-rendering” shows a low-level pattern that can be used to support efficient re-rendering in general, and which should be easy to extract into something higher level once we have made sure we have handled all of the necessary cases.

Previously, invoking a child template in a block helper looked like this: ```js registerHelper(‘identity’, function(params, hash, options, env) { return options.template.render(this, env, options, blockParams); }); ``` It now looks like this: ```js registerHelper(‘identity’, function(params, hash, options, env) { return options.template.render(this, blockParams); }); ```

In the future, frameworks like Ember that implement the `get` hook need some way to mark a render node as dirty. To that effect, we need to provide the `get` helper with a reference to its associated render node, such that if the underlying value changes, it can mark the node as dirty.

Previously, we would always revalidate all render nodes in a tree when calling `revalidate()` on the root render node. As a performance optimization, we introduce the notion of dirtiness to render nodes. We have updated all of the built-in hooks that modify render nodes to first check to see if the node is dirty. If the node is not dirty, we quickly bail out. When the render node related to a block helper (or component) is not dirty, it simply asks its child nodes to continue dirty checking. Just because a block helper or component is dirty does not necessarily mean that it needs to be replaced. For example, an `#if` helper whose content changes may not flip from the truthy template to the falsy template. For this reason, helpers can choose to return the result of rendering a new template, or communicate that the existing template is still stable. If a template is stable, HTMLBars treats it as if the node was not dirty to begin with, and continues dirty checking its children. To support this effort, all of the built-in hooks (content, block, subexpr, etc) now take a render node. These hooks can quickly check the render node for dirtiness and abandon any expensive work if it is false. The intent of this API is for consumers like Ember.js to narrowly target nodes for dirtying when using observers or streams, or dirty entire subtrees when triggered manually by users. Whenever a node becomes dirty, it would schedule a revalidation of the root render node. This guarantees that each node is checked only once, no matter how many changes occur in the tree during a single run loop. It also would guarantee that parent nodes have a chance to re-render themselves before any changes affecting their old children are processed.

This commit makes all tests pass with the new visitor structure, significantly shrinking the size of compiled templates and making it possible to do re-renders without having to execute a render function. This also means that, in theory, hooks like `inline` can know the structure of what they're invoking before actually invoking it, allowing implementation to do smarter caching at the "statement" level. This change is honestly mostly a wonky improvement, but it does help us avoid making as many decisions in the compiled template, deferring them to the runtime. There is still a decent amount of cruft left as of this commit, which we will remove momentarily.

Now that the template is largely just a data structure that describes its statements and expressions, the `render` function is no longer necessary. Instead, the runtime `render` function uses the data structure to decide what to do. At the moment, it simply delegates to the hooks that were already there, but the plan is to allow runtimes to completely swap out the expression visitor to do more exotic things like building and caching streams.

Previously, if a template’s first (or last) child was a dynamic node (like a mustache or block), the associated render node’s firstChild or lastChild would change. ```hbs {{#if condition}}{{value}}{{/if}} ``` In this case, the render node associated with the `#if` helper cannot have stable `firstNode` and `lastNode`, because when the nodes associated with `value` change, the nodes associated with the `if` helper must change as well. The solution was to require that templates create fragments with stable start and end nodes. If the first node is dynamic, we insert a boundary empty text node to ensure that the template’s boundaries will not change.

This commit refactors the internal scope handling of templates so that the concept of a scope, including block parameters, is rigorous. It also stops leaking the caller's scope into a helper call. This commit has two primary changes: * The internal "scope" is now broken up into `self` and `locals`. Block helpers now receive `options.template.yield`, which is used like `options.template.yield([ optional, block, params ])`. If block parameters are supplied, HTMLBars will ask the host to create a new scope frame, and the `locals` hash will be populated with the yielded parameters. It is still possible to shift the `self` by using `options.template.render(newSelf, [ optional, block, params ])`, which will always create a new scope frame. Since self-shifting helpers are intended to be rare, `yield` is the new path, and encapsulates all of the scoping details. * Helpers and block helpers no longer receive the `self` as `this`, because they are intended to be analogous to functions. Functions do not have access to the scope of the caller O_o. Note that hooks, which are used by the host to implement the scoping semantics, of course still have access to the scope.

This commit ensures that when any of `morph`, `env`, `scope`, `options`, or `blockArguments` exist in a function signature, that they appear in a consistent order (namely the one we used above). This commit also ends the practice of passing `env` to helpers, because helpers should be seen as functions operating within the scope system of HTMLBars, not arbitrary escape valves that can contort and abuse the template scope. Instead, environments implement the scoping system (and any keywords) by customizing the hooks invoked by template rendering. As of this commit, `partial` is a hook-defined keyword. In Ember `yield`, which also needs access to ambient scope information, will also be a hook-defined keyword. This finishes a separation between the hooks, which are used by environments to define the "programming language" of HTMLBars, and helpes, which are user-provided functions that operate within the scoping rules of HTMLBars.

```js registerHelper('random', function() { if (Math.random() > 0.5) { return this.yield(); } }); ``` This allows you to write a helper that takes a block and easily invokes it. The `yield` method can take block parameters: ```hbs {{#count as |i|}} <p>Rendered {{i}} times</p> {{/count}} ``` ```js var count = 0; registerHelper('count', function() { this.yield([ ++count ]); }); ``` Behind the scenes, this method passes along the entire current scope as well as the render node being filled in and the contextual element.

The RenderNode already has a reference to its contextual element, so passing extracting it from the render node is unnecessary.

mmun · 2015-03-29T18:48:54Z

Continued in #318.

tomdale force-pushed the reactive-rerendering branch from 95e6357 to 26541a0 Compare February 4, 2015 19:54

rwjblue reviewed Feb 6, 2015
View reviewed changes

chadhietala referenced this pull request in emberjs/ember.js Feb 9, 2015

Make Ember.HTMLBars version of registerHelper private.

ad62ccf

wycats force-pushed the reactive-rerendering branch from fd345f7 to 13257d7 Compare February 10, 2015 22:37

stefanpenner mentioned this pull request Feb 12, 2015

Cached computed properties seem to be invalidated too often emberjs/ember.js#10433

Closed

Tom Dale and Yehuda Katz and others added 19 commits February 12, 2015 11:26

Extract static cached fragment logic to runtime

d65b07c

Break apart hydration code into phases

a3d7112

Fix failing tests

2b0312f

renderNodes -> buildRenderNodes

1066f32

Too many hooks.

fe8096e

Guard if no parent and no contextualElement

97f0caf

Rename rerender to revalidate

de981e2

Propagate ownerNode through the render nodes

ff0dfea

Maintain consistent “shape” for options hash

61a2fe9

Tom Dale and Yehuda Katz and others added 4 commits February 12, 2015 11:28

Use extracted morph-range library

cf27996

Make ordering of scope/env consistent

5c06422

wycats force-pushed the reactive-rerendering branch from 769af95 to 5c06422 Compare February 12, 2015 19:29

Tom Dale and Yehuda Katz added 4 commits February 12, 2015 14:44

Hooks don't need to understand contextualElement

4ee6315

The RenderNode already has a reference to its contextual element, so passing extracting it from the render node is unnecessary.

Start writing host hook documentation

e836007

mixonic mentioned this pull request Feb 13, 2015

HTMLBars vs. Handlebars compiled output comparison emberjs/ember.js#10454

Closed

mmun mentioned this pull request Feb 19, 2015

[Question] Standalone usage (in combination with bound-templates.js?!) #301

Open

jayphelps mentioned this pull request Mar 20, 2015

[WIP] Globals + canary bower artifact repo deployment #300

Open

4 tasks

mmun closed this Mar 29, 2015

mmun mentioned this pull request Mar 30, 2015

Idempotent rerender #318

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Reactive rerendering #282

[WIP] Reactive rerendering #282

wycats commented Feb 3, 2015

wycats commented Feb 3, 2015

lin7sh commented Feb 3, 2015

cibernox commented Feb 5, 2015

mixonic commented Feb 5, 2015

stefanpenner commented Feb 5, 2015

cibernox commented Feb 5, 2015

rwjblue Feb 6, 2015

wycats Feb 6, 2015

mmun commented Mar 29, 2015

[WIP] Reactive rerendering #282

[WIP] Reactive rerendering #282

Conversation

wycats commented Feb 3, 2015

TODO

wycats commented Feb 3, 2015

lin7sh commented Feb 3, 2015

cibernox commented Feb 5, 2015

mixonic commented Feb 5, 2015

stefanpenner commented Feb 5, 2015

cibernox commented Feb 5, 2015

rwjblue Feb 6, 2015

Choose a reason for hiding this comment

wycats Feb 6, 2015

Choose a reason for hiding this comment

mmun commented Mar 29, 2015