I have a texture coord clamp solution working here right now, and it really works wonders for the clean appearance of scaled/rotated sprites. It's a pretty simple idea, even though nobody seems to be using it. I wonder why.
Here's what I'm doing:
struct VertexInput
{
float2 vertexPos : POSITION0; // from vertex data
float3 targetPos : TEXCOORD0; // everything else is from instancing data
float2 size : TEXCOORD1;
float4 texsource : TEXCOORD2; // texture atlas area, xy topleft, zw size
float4 colour : TEXCOORD3;
float2 param : TEXCOORD4; // x is rotation, yz is handle position
}
struct VertexOutput
{
float4 pos : POSITION0;
float4 colour : TEXCOORD2;
float2 tex : TEXCOORD0; // texture coord
float4 texArea: TEXCOORD1; // texture clamping area - xy topleft, zw size. We need to stay inside that to stay clear of interpolation artifacts
};
.....
// in the vertex shader, determine which scale is going to be used by dividing screen size by tex source size
float borderSize = min( inp.size.x / inp.texsource.z, inp.size.y / inp.texsource.w);
// round down to next power of two
borderSize = exp2( floor( log2( borderSize)));
// calculate the number of texels we need to stay away from the border. For highest mipmap level,
// 0.5 texels is enough to avoid interpolation artifacts, each further mipmap level needs twice that area
// but limit it to some sensible maximum
borderSize = 0.5f / clamp( borderSize, 1.0f/16.0f, 1.0f);
outp.texArea = (inp.texsource.xyxy + float4( 0, 0, 1, 1) * inp.texsource.zwzw + float4( 1, 1, -1, -1) * borderSize) * gInverseTextureSize.xyxy;
......
// then in the fragment shader, simply clamp the texture coordinate to the calculated texture area
// so that whatever fragment we render, interpolation never takes samples from outside the source texture area into account
float2 texk = clamp( inp.tex.xy, inp.texArea.xy, rein.texArea.zw);
float4 texel = tex2D( TexDiffuse, texk);
Quickly translated from my german shader code, so it might contain errors. But it's really that simple, I really wonder why nobody else is using it. I draw tens of thousands of sprites with this, arbitrarily scaled, rotated, positioned with subpixel accuracy, and there are no interpolation artifacts visible. The downside is that you'll need to come up with a better mipmap level calculation - the one given above is obviously aimed at 2D sprites only. But I'm pretty sure you can find some calculation that safely overestimates the mipmap level, especially when you're not bound to DX9 shaders like I am currently.
For starters, you can simply leave out the mipmap level calculation and use a fixed 0.5 texel offset from the border. That way you should be able to use at least bilinear filter and/or AntiAliasing. MipMapping will still come up with artifacts, though.
Developer
