Writing a plugin for upscaling rendering under ThreeJS

Introduction

An upscaling algorithm is an algorithm for improving the quality of real-time animations and images using simple and efficient shaders. Unlike AMD FSR, which is generally focused on PCs and consoles, we will create a plugin that has been designed specifically for upscaling animations, preserving clarity and contrast while working on all platforms.

To implement the plugin in Three.js based on the idea of upscaling, we need to create a shader that will apply filtering and image enhancement in real time. The plugin will work cross-platform thanks to WebGL.

You can view the full source of this tutorial at GitHub: https://github.com/DevsDaddy/threejs-upscaler

Step 1: Create a shader for upscaling

We need to create a shader that implements the basic upscaling principles, such as contour enhancement and border smoothing, to increase the clarity of the image.

Here is an example of a simple shader in GLSL:

// Vertex Shader
varying vec2 vUv;

void main() {
    vUv = uv;
    gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}

// Fragment Shader
uniform sampler2D tDiffuse;
uniform vec2 resolution;

varying vec2 vUv;

void main() {
    vec2 texelSize = 1.0 / resolution;

    // Get neighbor pixels
    vec4 color = texture2D(tDiffuse, vUv);
    vec4 colorUp = texture2D(tDiffuse, vUv + vec2(0.0, texelSize.y));
    vec4 colorDown = texture2D(tDiffuse, vUv - vec2(0.0, texelSize.y));
    vec4 colorLeft = texture2D(tDiffuse, vUv - vec2(texelSize.x, 0.0));
    vec4 colorRight = texture2D(tDiffuse, vUv + vec2(texelSize.x, 0.0));

    // Work with edges
    float edgeStrength = 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorUp.rgb));
    edgeStrength += 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorDown.rgb));
    edgeStrength += 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorLeft.rgb));
    edgeStrength += 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorRight.rgb));
    edgeStrength = clamp(edgeStrength, 0.0, 1.0);

    // Apply filtering
    vec3 enhancedColor = mix(color.rgb, vec3(1.0) - (1.0 - color.rgb) * edgeStrength, 0.5);
    gl_FragColor = vec4(enhancedColor, color.a);
}

This shader enhances object boundaries by increasing the contrast between neighboring pixels, which creates a smoothing effect and enhances details.

Step 2: Create a plugin for Three.js

Now let's integrate this shader into Three.js as a plugin that can be used to upscale the scene in real time.

import * as THREE from 'three';

// Simple Upscaling Plugin
class UpscalerPlugin {
    constructor(renderer, scene, camera, options = {}) {
        this.renderer = renderer;
        this.scene = scene;
        this.camera = camera;

        // Plugin Settings
        this.options = Object.assign({
            useEdgeDetection: true,  // Edge Detection
            scaleFactor: 2.0,        // Upscaling Value
        }, options);

        this.onSceneDraw = null;
        this.onRender = null;

        this.initRenderTargets();
        this.initShaders();
    }

    initRenderTargets() {
        // Create render textures
        const renderTargetParams = {
            minFilter: THREE.LinearFilter,
            magFilter: THREE.LinearFilter,
            format: THREE.RGBAFormat,
            stencilBuffer: false,
        };

        const width = this.renderer.domElement.width / this.options.scaleFactor;
        const height = this.renderer.domElement.height / this.options.scaleFactor;

        this.lowResTarget = new THREE.WebGLRenderTarget(width, height, renderTargetParams);
        this.highResTarget = new THREE.WebGLRenderTarget(width * this.options.scaleFactor, height * this.options.scaleFactor, renderTargetParams);
    }

    // Init Upscaling Shaders
    initShaders() {
        this.upscalerShader = {
            uniforms: {
                'tDiffuse': { value: null },
                'resolution': { value: new THREE.Vector2(this.renderer.domElement.width, this.renderer.domElement.height) },
            },
            vertexShader: `
                varying vec2 vUv;
                void main() {
                    vUv = uv;
                    gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
                }
            `,
            fragmentShader: `
                uniform sampler2D tDiffuse;
                uniform vec2 resolution;

                varying vec2 vUv;

                void main() {
                    vec2 texelSize = 1.0 / resolution;

                    // Get Neighbor Pixels
                    vec4 color = texture2D(tDiffuse, vUv);
                    vec4 colorUp = texture2D(tDiffuse, vUv + vec2(0.0, texelSize.y));
                    vec4 colorDown = texture2D(tDiffuse, vUv - vec2(0.0, texelSize.y));
                    vec4 colorLeft = texture2D(tDiffuse, vUv - vec2(texelSize.x, 0.0));
                    vec4 colorRight = texture2D(tDiffuse, vUv + vec2(texelSize.x, 0.0));

                    // Work with edges
                    float edgeStrength = 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorUp.rgb));
                    edgeStrength += 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorDown.rgb));
                    edgeStrength += 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorLeft.rgb));
                    edgeStrength += 1.0 - smoothstep(0.1, 0.3, length(color.rgb - colorRight.rgb));
                    edgeStrength = clamp(edgeStrength, 0.0, 1.0);

                    // Applying edges incresing and filtering
                    vec3 enhancedColor = mix(color.rgb, vec3(1.0) - (1.0 - color.rgb) * edgeStrength, 0.5);

                    gl_FragColor = vec4(enhancedColor, color.a);
                }
            `
        };

        this.upscalerMaterial = new THREE.ShaderMaterial({
            uniforms: this.upscalerShader.uniforms,
            vertexShader: this.upscalerShader.vertexShader,
            fragmentShader: this.upscalerShader.fragmentShader
        });

        // Generate Sample Scene
        if(!this.onSceneDraw){
            this.fsQuad = new THREE.Mesh(new THREE.PlaneGeometry(2, 2), this.upscalerMaterial);
            this.sceneRTT = new THREE.Scene();
            this.cameraRTT = new THREE.OrthographicCamera(-1, 1, 1, -1, 0, 1);
            this.sceneRTT.add(this.fsQuad);
        }else{
            this.onSceneDraw();
        }
    }

    render() {
        // Render scene at low resolution
        this.renderer.setRenderTarget(this.lowResTarget);
        this.renderer.render(this.scene, this.camera);

        // apply upscaling
        this.upscalerMaterial.uniforms['tDiffuse'].value = this.lowResTarget.texture;
        this.upscalerMaterial.uniforms['resolution'].value.set(this.lowResTarget.width, this.lowResTarget.height);

        // Render to window
        this.renderer.setRenderTarget(null);
        if(!this.onRender)
            this.renderer.render(this.sceneRTT, this.cameraRTT);
        else
            this.onRender();
    }
}

export { UpscalerPlugin };

Step 3: Using the plugin in the project

Now let's integrate the plugin into our Three.js project.

import * as THREE from 'three';
import { UpscalerPlugin } from './upscaler.js';

// Create Renderer
const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

// Create Three Scene
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
camera.position.z = 5;

// Create Geometry
const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);

// Initialize Our Upscaler Plugin
const upscaler = new UpscalerPlugin(renderer, scene, camera, {
    scaleFactor: 1.25,
    useEdgeDetection: true
});

// Initialize stats monitor
const stats = new Stats();
stats.showPanel(0); // 0: FPS, 1: MS, 2: MB
document.body.appendChild(stats.dom);

// On Window Resizing
function onWindowResize() {
    renderer.setSize(window.innerWidth, window.innerHeight);
    camera.aspect = window.innerWidth / window.innerHeight;
    camera.updateProjectionMatrix();

    // Update Upscaler Parameters
    upscaler.initRenderTargets();
    upscaler.initShaders();
}

window.addEventListener('resize', onWindowResize, false);

function animate() {
    stats.begin();

    requestAnimationFrame(animate);

    // Animate Sample Cube
    cube.rotation.x += 0.01;
    cube.rotation.y += 0.01;

    // Render with upscaling
    upscaler.render();

    stats.end();
}

animate();

Step 4: Cross-platform support

The plugin uses WebGL, which is supported by most modern browsers and devices, making it cross-platform. It will work correctly on both desktops and mobile devices.

Conclusion

This plugin for Three.js provides real-time scene upscaling using GPU shader processing approach. It improves contrast and border clarity, which is especially useful for animated scenes or low-resolution renders. The plugin integrates easily into existing projects and provides cross-platform performance by utilizing WebGL.

You can view the full source at GitHub:

https://github.com/DevsDaddy/threejs-upscaler

---

You can also help me out a lot in my plight and support the release of new articles and free for everyone libraries and assets for developers:

My Discord | My Blog | My GitHub

BTC: bc1qef2d34r4xkrm48zknjdjt7c0ea92ay9m2a7q55
ETH: 0x1112a2Ef850711DF4dE9c432376F255f416ef5d0
USDT (TRC20): TRf7SLi6trtNAU6K3pvVY61bzQkhxDcRLC