— Pixels Commander

[ In English, На русском ]

HTML/CSS rendering via WebGL for highest performance possible and unlimited animations abilities on the Web

In recent time Web development community had a big discussion on “DOM is slow” topic. This thesis is truthful. DOM is a quite complex model which starts a ripple of events or chain reaction over document on every modification. This impacts animations first of all.  Since desktop browsers are mostly fine with handling animations at 60 FPS, mobile and embedded devices still provide bad, janky user experience.

Finding solution for this problem is a high priority issue for contemporary Web development. Active researches in this field are ongoing for a long time (got additional impulse in 2011 after Facebook refused hybrid approach), however first systematic solutions were developed just recently. These solutions could be divided into two subsets:

  • Acceleration of classic DOM operations;
  • Alternative content rendering technologies.

And if first branch have already reached the limit of contemporary browsers API abilities, alternative content rendering technologies approach still have a lot of untapped potential. One of the most significant achievements in the field of No DOM web development is React-Canvas. As we see from naming it uses Canvas API to render content allowing to build layouts by using specific React components: <Surface>, <Layer>, <Group>, <Text>, <Image>, <ListView>. This approach shows nice performance, but also have cons which are “incompatible with life”. Let`s imagine ideal solution using alternative rendering technology:

  • Best solution will be framework agnostic and look more like a polyfill for smooth animations problem;
  • It will not force developer to study new conceptions and keep him developing in common HTML/CSS environment;
  • Game engines world passed Canvas vs WebGL fight few years ago. WebGL is treated as more performant so ideal 60FPS solution will use WebGL with Canvas fallback;
  • Best solution will keep DOM tree since it allows more effective debugging and application state monitoring using Dev Tools.

Our research on “slow DOM problem” took theses listed above as an input. After two years of experiments we got HTML GL – library which allows to render HTML/CSS content in WebGL. It is easy to use, framework agnostic and does not require developer to study new concepts or technologies keeping him in common HTML/CSS/JS world.

 

How to use HTML GL?

GitHub repository

Include htmlgl.js into your project. Use tag name <html-gl> for elements you are going to animate. These elements will be rendered in WebGL and their CSS Transform property will be a facade for WebGL representation transformations. The DOM element itself will not be displayed or animated. All transforms happens on GPU and affect only WebGL textures representing element. This decreases resources consuming and allows better control over resources usage. What differentiate HTML GL  from React-canvas or similar is that it keeps hidden but still actual DOM structure and is easier to debug.

 

Demos

  • FX Demo WebGL is not only about performance, it breaks web interactivity limits
  • Basic HTML GL demonstrates HTML GL based on simple content + smooth animations with Velocity.js and transformations via CSS Transform
  • Basic DOM the same project with HTML GL disabled, so all animations run on CSS animations and DOM nodes. It is easy to see that on the peak of animations layer is being repainted, which means jank on mobile device. HTML GL version do not have this issue
  • Advanced content HTML GL slider with nested structure, rendered via HTML GL and animated with Velocity.js
  • Advanced content DOM

 

Under the hood

Main idea behind HTML GL is controllable HTML/CSS rasterization and uploading it to GPU as a texture. It is similar to hardware accelerated CSS, but have direct output to the screen and we have more control over.

  • html-gl was created on the page
  • Elements content is being rasterized
  • Then displayed on fullscreen WebGL context as a 2d sprite, the DOM element itself hides
  • style.transform mapped to it`s WebGL representation and modifies WebGL representation
  • If content of html-gl node was chaged HTML GL update texture (based on DOM Mutation Observers / Events) and update WebGL texture

Conclusion

As you see the flow allows to mix classic DOM content and HTML GL in one application. HTML GL do not limit user in choosing a framework, do not force to study specific APIs. Just add <html-gl> to tweak performance and be able to add amazing effects.

Talk slides

  • Maxim Kravchenko

    Доклад на кодефесте взорвал голову, спасибо! =)
    Если привлекать к рендерингу WebGL, то это означает что JavaScript работает 60 раз в секунду, а следовательно, ради плавности, тяжелую логику в отклик на действия пользователя уже позволить нельзя.
    Понятное дело, что если использовать эту вещь в меру, то все обойдется, а если нужно чтобы некий эффект как в демке рендерился постоянно, то наверное придется ради профита отказаться от Event Driven в пользу отработки логики приложения во время прохода rendering loop. Все-таки JS однопоточный и это в некоторой степени ограничивает в использовании таких клёвых эффектов. Не приходили такие мысли?

  • Anonymous

    Тут важно понимать, что эффекты рендерятся на GPU (пиксельные и вершинные шейдеры). Это значит, что на процессор и главный поток браузера они почти не влияют, поэтому без паники! Просто попробуй написать GL SL или взять с shadertoy.com и запплаить на свой HTML GL.

    Important thing here is that effects are GPU rendered (pixel and vertex shaders). It means that it do not affect main thread too much. I would not expect unsolvable performance issues here. So no worries, just try to write or copy/paste from shadertoy.com some GL SL and apply to your HTML GL. Cheers!

  • woot

    Sorry, I don’t think this makes sense. You should rather try to implement a better backend for WebKit.

  • Anonymous

    The sense is to be able to do like http://pixelscommander.com/polygon/htmlgl/demo/filters.html . WebKit is almost dead BTW

  • http://bucaran.me Jorge Bucaran

    Wow, what do you mean by WebKit is almost dead?

  • Anonymous

    Of course it is not literally dead, but saying “better WebKit will save humanity” sounds a bit weird since more and more projects move to Chromium and Blink right now. WebKit need a good kick to take the lead again.

  • Kyrre Eilertsen

    Eh.. chromium is webkit..

  • bob

    not being able to select text is deal-breaker

  • Anonymous

    There are workarounds to implement selection and we are going to do that. Follow repository on github, there should be an update.

  • tips4design

    Can we use this to make snapshots of websites, simillar to what HTML2CANVAS (ttps://github.com/niklasvh/html2canvas) does?

  • Anonymous

    No, html2canvas is the best approach you can get right now. But if you would like to have better one you definitely should read this http://pixelscommander.com/en/javascript/state-of-html-content-rasterization-draw-html-to-canvas-image/ and spread the word about Rasterization API proposed in the end.

  • David Spector

    As I understand it, this library automatically caches the image of a web page, or part of one, even if the DOM is dynamically changed over time, and automatically displays the cached image instead of the rendered DOM, permitting simple high-speed animations of the image that would not otherwise (at least on a mobile platform) be possible (with the limitation that magnifying the image necessarily introduces blur). In my opinion, this is an elegant solution to a common problem that deserves to be widely known.

  • Anonymous

    Thanks, David. This is absolutely correct.

    HTML GL could be even faster if we have native Rasterization API. The problem is described here http://pixelscommander.com/en/javascript/state-of-html-content-rasterization-draw-html-to-canvas-image/ and this is the proposal draft https://gist.github.com/PixelsCommander/a0b5882139cbb8a1781c#file-proposal-md. You may help to make browsers better and HTML GL faster by spreading word about Rasterization API. The aim is to start discussion on this topic and make it visible for w3c group.

  • Max

    I’ve tried to make a demo with your advice from how to use section and it didn’t working unfortunately.

    all it does is adds opacity: 0; and some styles to html-gl tag

  • Anonymous

    Could you share code?

  • Zdzichu

    The text in the Basic HTML GL demo is unbearably blurry. I’m using a normal 1080p monitor, not Retina. So it’s rather useless for text rendering and it has nothing to do with Retina.

  • Anonymous

    It is a bug for your particular case and it will be solved if have more info. What was the screen resolution and window size?

  • Anonymous

    It is a bug for your particular case and it will be solved if have more info. What was the screen resolution and window size?

  • Jay

    Why don’t you use distance fields to render fonts? Wouldn’t that solve your “blurry” problem?

  • Anonymous

    Are you a Retina user? Retina compatibility was fixed few days ago but not reflected in demos yet. Check out recent htmlgl from github. Not sure what do you mean by solving blurry with distance fields could you elaborate please?

  • Jay

    I may be misunderstanding how you’re rendering the html elements to a texture. I haven’t read the code, but if you’re rendering the text in htmlgl’s JavaScript using WebGL, and you are not somehow using the browser itself to render text, then distance fields can be used as a GPU font cache format that allows for perfect scaling: http://youtu.be/CGZRHJvJYIg

  • Anonymous

    Ah, got it. You mean Valve`s SDF. Not using it yet, but agree it is the most advanced way to render fonts in WebGL

  • http://workingdraft.de/237/ Revision 237: Remote Work und Informationsnachschub | Working Draft

    […] HTMLGL […]

  • Anonymous

    Got any links to where this is being used in production?

  • Anonymous

    Seems like devicePixelRatio isn’t being handled, as on my Retina screen the rendering is fuzzy instead of crisp. Would be nice to see that working. 🙂

  • Anonymous

    Here’s a screenshot, the WebGL part fuzzy.

  • Anonymous

    Oh!! Hehe. I didn’t read the TODO part before posting. x} Looking forward to it. It’s not a hard fix, just simply factor the device pixel ratio. It’s like 2 lines of code.

    EDIT: Well, maybe not 2 lines, but it’s simple: https://www.khronos.org/webgl/wiki/HandlingHighDPI

  • Anonymous

    No problem, actually retina issue is fixed in recent version which is on GitHub

  • Anonymous

    Oh!! Hehe. I didn’t read the TODO part before posting. x} Looking forward to it. It’s not a hard fix, just simply factor the device pixel ratio. It’s like 2 lines of code.

    EDIT: Well, maybe not 2 lines, but it’s simple: https://www.khronos.org/webgl/wiki/HandlingHighDPI

  • Anonymous

    Oh!! Hehe. I didn’t read the TODO part before posting. x} Looking forward to it. It’s not a hard fix, just simply factor the device pixel ratio. It’s like 2 lines of code.

    EDIT: Well, maybe not 2 lines, but it’s simple: https://www.khronos.org/webgl/wiki/HandlingHighDPI

  • Anonymous

    Here’s a screenshot, the WebGL part fuzzy.

  • Anonymous

    Here’s a screenshot, the WebGL part fuzzy.

  • Anonymous

    Seems like devicePixelRatio isn’t being handled, as on my Retina screen the rendering is fuzzy instead of crisp. Would be nice to see that working. 🙂