Image manipulation and texture mapping using HTML5 Canvas?

I think you will never get an accurate result… I spent some time investigating how to do 3d graphics using canvas 2d context and I found it viable to do texture mapping gouraud shading by computing appropriate 2d gradients and matrices:

  • Solid polygons are of course easy
  • Gouraud filling is possible only on one component (i.e. you cannot have a triangle where every vertex is an arbitrary RGB filled with bilinear interpolation, but you can do that filling using for example three arbitrary shades of a single color)
  • Linear texture mapping can be done using clipping and image drawing

I would implement perspective-correct texture mapping using mesh subdivision (like on PS1).

However I found many problems… for example image drawing with a matrix transform (needed for texture mapping) is quite inaccurate on chrome and IMO it’s impossible to get a pixel-accurate result; in general there is no way to turn off antialiasing when drawing on a canvas and this means you will get visible see-through lines when subdividing in triangles. I also found multipass rendering working really bad on chrome (probably because of how hw-accellerated rendering is implemented).

In general this kind of rendering is surely a stress for web browsers and apparently these use cases (strange matrices for example) are not tested very well. I was even able to get Firefox crashing so bad that it took down the whole X susbsystem on my Ubuntu.

You can see the results of my efforts here or as a video here… IMO is surely impressing that this can be done in a browser without using 3D extensions, but I don’t think current problems will be fixed in the future.

Anyway the basic idea used to draw an image so that the 4 corners ends up in specific pixels position is to draw two triangles, each of which will use bilinear interpolation.

In the following code I assume you have a picture object texture and 4 corners each of which is an object with fields x,y,u,v where x,y are pixel coordinates on the target canvas and u,v are pixel coordinates on texture:

function textureMap(ctx, texture, pts) {
    var tris = [[0, 1, 2], [2, 3, 0]]; // Split in two triangles
    for (var t=0; t<2; t++) {
        var pp = tris[t];
        var x0 = pts[pp[0]].x, x1 = pts[pp[1]].x, x2 = pts[pp[2]].x;
        var y0 = pts[pp[0]].y, y1 = pts[pp[1]].y, y2 = pts[pp[2]].y;
        var u0 = pts[pp[0]].u, u1 = pts[pp[1]].u, u2 = pts[pp[2]].u;
        var v0 = pts[pp[0]].v, v1 = pts[pp[1]].v, v2 = pts[pp[2]].v;

        // Set clipping area so that only pixels inside the triangle will
        // be affected by the image drawing operation
        ctx.save(); ctx.beginPath(); ctx.moveTo(x0, y0); ctx.lineTo(x1, y1);
        ctx.lineTo(x2, y2); ctx.closePath(); ctx.clip();

        // Compute matrix transform
        var delta = u0*v1 + v0*u2 + u1*v2 - v1*u2 - v0*u1 - u0*v2;
        var delta_a = x0*v1 + v0*x2 + x1*v2 - v1*x2 - v0*x1 - x0*v2;
        var delta_b = u0*x1 + x0*u2 + u1*x2 - x1*u2 - x0*u1 - u0*x2;
        var delta_c = u0*v1*x2 + v0*x1*u2 + x0*u1*v2 - x0*v1*u2
                      - v0*u1*x2 - u0*x1*v2;
        var delta_d = y0*v1 + v0*y2 + y1*v2 - v1*y2 - v0*y1 - y0*v2;
        var delta_e = u0*y1 + y0*u2 + u1*y2 - y1*u2 - y0*u1 - u0*y2;
        var delta_f = u0*v1*y2 + v0*y1*u2 + y0*u1*v2 - y0*v1*u2
                      - v0*u1*y2 - u0*y1*v2;

        // Draw the transformed image
        ctx.transform(delta_a/delta, delta_d/delta,
                      delta_b/delta, delta_e/delta,
                      delta_c/delta, delta_f/delta);
        ctx.drawImage(texture, 0, 0);
        ctx.restore();
    }
}

Those ugly strange formulas for all those “delta” variables are used to solve two linear systems of three equations in three unknowns using Cramer’s method and Sarrus scheme for 3×3 determinants.

More specifically we are looking for the values of a, b, … f so that the following equations are satisfied

a*u0 + b*v0 + c = x0
a*u1 + b*v1 + c = x1
a*u2 + b*v2 + c = x2

d*u0 + e*v0 + f = y0
d*u1 + e*v1 + f = y1
d*u2 + e*v2 + f = y2

delta is the determinant of the matrix

u0  v0  1
u1  v1  1
u2  v2  1

and for example delta_a is the determinant of the same matrix when you replace the first column with x0, x1, x2. With these you can compute a = delta_a / delta.

Leave a Comment