p5 loadFont() and its dilemma with opentype.js

[munusshih]

Topic

As @amehowc mentioned in his p5.varaibleFont library
There are two primary methods for loading and using fonts in p5.js. One involves using loadFont(), and the other is by defining a font either through CSS with a @font-face tag or importing it via an API like the Google Fonts API. Both methods have the same end result when using text() in the 2D context. However, differences emerge when you delve into font data functions like font.textToPoints() or font.getBounds(), or when using the WEBGL context.

Basically it could be drawn into a table like this:

Loading Through	WEBGL text()	2D text()	variable font using CSS variation-font-settings	native Canvas API	p5.font (like textToPoint)
loadFont(), relying on opentype.js	O	O	X	X	O
CSS @font-face (such as google fonts)	X	O	O	O	X

O = supported feature, X = unsupported feature

The two ways of loading fonts create different supports in p5 for these typographic related functions.

Supports for native Canvas API

• If you rely on using opentype.js to parse through the font, you get all the vertices of the font, and can draw them even in WEBGL mode.
• However, you will lose important supports like controlling the texts with native Canvas API (because Canvas doesn't recognize this as text anymore, but vertices with text shapes). This is really major because p5 inherently is built on top of HTML Canvas, which is a relatively new element in web technologies. Many things that have long been requested by the community are slowly being added in, such as letter-spacing or fontStretch. We might be reinventing the wheels here if we decide to not have it support native Canvas APIs.

The loadingFont function

Basically if you import font using the loadFont() function, it will still create a CSS @font-face for you in the code as shown here in loading_displaying.js:

p5.js/src/typography/loading_displaying.js

Lines 125 to 133 in 36437b3

	if (validFontTypes.includes(fileExt)) {
	fontFamily = fileNoPath.slice(0, lastDotIdx !== -1 ? lastDotIdx : 0);
	newStyle = document.createElement('style');
	newStyle.appendChild(
	document.createTextNode(
	`\n@font-face {\nfont-family: ${fontFamily};\nsrc: url(${path});\n}\n`
	)
	);
	document.head.appendChild(newStyle);

However, it will not be referencing it, so I'm really unsure if this part of the code is even working.

[welcome]

Welcome! 👋 Thanks for opening your first issue here! And to ensure the community is able to respond to your issue, please make sure to fill out the inputs in the issue forms. Thank you!

[Qianqianye]

Thanks @munusshih. I'm inviting some typography contributors to this discussion @dhowe @kyeah @aahdee @limzykenneth @davepagurek

[davepagurek]

What do X and O represent in the table? If O is a supported feature, should the top right cell of the table be an O?

This is a tough one. If we rely only on js canvas, then we get all the features it has, but it also prevents us from extending those features at all, since it doesn't give us access to the underlying data. I don't think there's a way we can support textToPoints or WebGL text without using a separate implementation unfortunately, so while the library still aims to provide support for those, we need to rely on something else. I think it still makes sense to want to support those APIs -- the Generative Design book has a lot of interesting examples of using textToPoints to make cool art, and it feels like a benefit of using p5 rather than just using the native canvas methods directly.

That said, we don't want to reinvent the wheel, especially for something as complex as font layout (the fact that it's so hard to find a JS library that does everything we need is maybe a testament to how hard it is!) Maybe another option is to try to rely on a more battle-tested library from outside of the Javascript ecosystem, and compile it to JS via emscripten? Something like Pango? Building a C library for JS is a big task in itself, but that might still be easier than patching variable font support AND layout support into opentype.js and p5.

[munusshih]

@davepagurek Edited! Thanks for pointing that out. Got a little confused in making the table in markdown.

I was originally thinking of having the two system co-existing like what we have right now, but prioritize CSS@font-face as the main rendering way for 2D texts to better support the HTML Canvas API, and have opentype.js stand as an support system for WEBGL and p5.font solutions.

This however, might be a BIG change in code as it is the opposite logic now (prioritize opentype.js and have CSS@font-face as a supportive system).

[dhowe]

Not a trivial problem (the two font-handling pathways have always bothered me). If we had a way to handle text in opengl, I might arguing for moving p5.font to an external library, for the sake of simplicity. In any case perhaps it would be helpful to spec out the 'must-have' and 'would-be-nice' functionality...

[davepagurek]

If we had a way to handle text in opengl, I might arguing for moving p5.font to an external library

To clarify, would that be for "extra" things like textToPoints and reading text paths, and just leave text rendering in the core library? That could make sense, although unfortunately it's a big "if", since WebGL kinda leaves us on our own for text rendering, so we're reliant on getting the paths to do any rendering at all. Some alternative approaches could be to use a 2D canvas to render text to textures that WebGL uses, but it would probably mean having an involved caching system to support arbitrary zooms without having blurry text or using too much memory.

the two font-handling pathways have always bothered me

Yeah, I feel like having two code paths for such a complex system might open us up to more bugs once we start adding more things like variables and layout into the mix. If it's possible to use a library that does everything we need, it would be nice to load a default font if unspecified and use the same system everywhere. But that's also a big "if" :')

[arthurcloche]

I'm not a seasoned contributor, so I might miss some points, but I'm happy to share my thoughts.

Some comments on your table @munusshih:

Loading font using loadFont() prevents the use of the variable-axis and native API
That's not entirely true. The tweak to add a proper CSS name to a loaded font is minimal since, as you mentioned, we create a CSS font tag for it already. In 2D, whether you use CSS or p5.Font, the font is applied the same way using the webCanvas's drawingContext.font. All webCanvas functionalities are available, and most of them are also feasible in WebGL. Given the way the fonts are rendered, attributes like wordSpacing or textStretch are not too complex. You just have to translate or scale the geometries the letters are rendered into.

Access a Google Font with opentype.js
This issue might be resolved with a more customizable font loader. I've built one for my own needs, modeling it after the webFont loader. For cases where we need the file, e.g., for textToPoints, the Google API uses static files for their library. These are added as a CSS file with a URL. With some query magic, it's not hard to retrieve that URL and use it in loadFont().

Some direction might be found in this proposition for glyph support (though it might be outdated):
https://docs.google.com/document/d/1hWSmdaNKdt7grYK_XJ3wu601kysYewZcPkqxBvTYSX8/edit#heading=h.a93rkf3s2i24

And I didn't have the time to explore fully, but it seems there might be clues on how to load system fonts as font data here:
https://developer.chrome.com/en/articles/local-fonts/

About opentype.js
We use opentype.js to retrieve the characters' paths as Bezier points for later use. The rest of the font rendering or data-handling is executed within the p5 scope. The function textToPoints is akin to SVG's pointAtLength, while WEBGL is a tad more intricate. If we can access the font data on our end, we won't need to rely on opentype.js any longer, and all font functions would remain operational. Many type visualization apps showcase advanced capabilities in accessing a font file, such as Font Gauntlet by Dinamo. And since @davepagurek mentioned porting some C to JS, we could also peek in the direction of Python. After all, it's the most commonly used language for designing fonts nowadays, with a well-known library being FontTools.

[arthurcloche]

To clarify, would that be for "extra" things like textToPoints and reading text paths, and just leave text rendering in the core library? That could make sense, although unfortunately it's a big "if", since WebGL kinda leaves us on our own for text rendering, so we're reliant on getting the paths to do any rendering at all. Some alternative approaches could be to use a 2D canvas to render text to textures that WebGL uses, but it would probably mean having an involved caching system to support arbitrary zooms without having blurry text or using too much memory.

And what about geometries ? Since we get all we need to make it a mesh, how expensive would it be to have characters being p5.Geometry objects or vertex buffer arrays ? Inspiration might be three.js

[davepagurek]

TIL about the SFNT data access in the local fonts API! That's pretty cool, and could let us do what opentype.js lets us do but for local fonts. I hope they add to the spec to give that same access for web fonts.

And what about geometries ? Since we get all we need to make it a mesh, how expensive would it be to have characters being p5.Geometry objects or vertex buffer arrays?

We actually do something like this at Butter now! The main issue we had to work around is that the level of vertex detail you want depends on both the size of the text, and how far from the camera it is, so a single p5.Geometry glyph may not look good if you move closer to it. We ended up with a caching system, but we also don't give users full camera control, which bounds the complexity a bit. For something like p5 we'd maybe need to make a version of p5.Geometry which knows how to dynamically regenerate itself for different scales, which could be a feasible solution instead of shaders. It'd probably still need to know about the bezier paths for all the glyphs though, so we'd still need opentype.js for web font support.

Also, one other idea: In the initial implementation PR for the WebGL font shader, there was some discussion of SDF fonts. They're less accurate than just using the fonts' bezier paths, but could plausibly be generated from 2D canvas renderings of glyphs without ever looking at the font data.

[arthurcloche]

Also, one other idea: In the initial implementation PR for the WebGL font shader, there was some discussion of SDF fonts. They're less accurate than just using the fonts' bezier paths, but could plausibly be generated from 2D canvas renderings of glyphs without ever looking at the font data.

Yeah, that's also a possible solution. I've been documenting this for a later attempt but never really delved into it.
Some interesting stuff about that are referenced in those :
https://github.com/mattdesl/text-modules
https://github.com/Jam3/three-bmfont-text

[davepagurek]

Just to enumerate the ideas we've discussed so far, here's a new pros/cons list.

If we make SDFs of text:

• 🟢 We remove the opentype.js dependency
• 🟢 We could use whatever 2D canvas APIs are available when generating glyphs, including variable font support, and anything added in the future without updating our WebGL rendering system
• 🟢 We keep WebGL support for fonts regardless of scale
• 🟡 We won't exactly match the appearance of fonts between WebGL and 2D mode (2D mode still being exact)
• 🟡 Generating an SDF when a variable is changed might be too slow to animate variable font changes? Need to experiment with that
• 🟡 We can still support textToPoints and textBounds, but built ourselves using via marching squares on the SDF, which also won't exactly match the 2D renderer's version
• 🟡 We have to rewrite our WebGL text rendering (+ textToPoints and textBounds)
• 🔴 Does not solve text layout issues, we still need to do that ourselves

If we find a replacement library (e.g. Pango, which supports layout and glyph info):

• 🟢 We remove the opentype.js dependency
• 🟢 We get variable font support
• 🟢 We get layout methods we can use
• 🟢 We can use the existing WebGL Bezier drawing shader for fonts
• 🟢 We get enough font info to keep textToPoints basically the same
• 🟡 We have to cross-compile another library to JS, potentially a big task, and potentially adding a lot to our bundle size
• 🔴 We would likely use this in both 2D and WebGL modes, not supporting 2D canvas APIs
• 🔴 We would not automatically get new 2D canvas typography features

If we support two systems:

• 🟢 We get full 2D canvas support
• 🟡 We have to implement our own WebGL support for every 2D canvas API
• 🟡 We have two code paths for two systems, so we likely need some new abstractions and testing to keep this maintainable
• 🟡 When new 2D canvas APIs are added, WebGL will not be able to use them until we implement them ourselves
• 🔴 Does not solve text layout issues, we still need to do that ourselves

[dhowe]

🟡 We have to cross-compile another library to JS, potentially a big task, and potentially adding a lot to our bundle size

do we currently do this with another library ?

[davepagurek]

@dhowe not yet! there are other libraries on GitHub we could look at for guidance, but it'd be something new for us.

[dhowe]

right, wasn't sure if I had missed this... anyhow, I'd caution against such a path (Pango or a similar port), unless there really are no other options -- the added complexity can be significant