next »

[st33d]

In this thread I will be exposing my research benchmarking Stage3D for Flash (I've been tasked at Nitrome with finding out the truth of the matter). Most importantly I need to test tablet and mobile performance. I will also be testing in comparison to my own blitting methods which generally manage 1000s of sprites at a time.

The Starling framework benchmark demo comes up with 760 objects on my desktop PC at work. That's shit.

I intend to find out what Stage3D can actually do without any Adobe bullshit hype. Empirical data that can be questioned.

I've got an iPad 2, if you've got other devices you could help out with, please let me know. I'll be starting research proper on Monday.

[JMickle]

I have a htc wildfire if you want any tests doing on that. i also have a crappy netbook if that's any use?

[Fallsburg]

Better tests:
ND2D, specifically the sprite cloud, sprite batch test (depending on whether the CPU or GPU is better on your computer one of those will be better).

simppa.fi -- Not a very realistic benchmark, but it shows what can be done.


Anyway, I think that straight blitting isn't a fair comparison, or at least not the only fair comparison.  To truly be fair, you should include alpha and rotation, things that straight blitting can't do well.

I'd envision a test set up of something like:
Dry | Alpha | Rotation | Rotation+Alpha
with maximum number of sprites that still gets some number of frames per second.  That way people can compare based on what they will be using.

[Franklin's Ghost]

I'm interested to see your results cause was thinking of trying starling for my next project.

I've got a samsung galaxy if you need some more devices to test on.

[st33d]

I'd envision a test set up of something like:
Dry | Alpha | Rotation | Rotation+Alpha
with maximum number of sprites that still gets some number of frames per second.  That way people can compare based on what they will be using.

Agreed. I'll be aiming to get a demo that can hold out at 30fps.

It really pissed me off when augmented reality was was claimed as being fast in Flash when I could only get a maximum of 15fps using an optimised library and the lowest resolution markers. 30fps or bust, it's the standard for iOS anyway.

[Chromanoid]

I'm looking forward to your results.

I guess you already read the optimization hints here http://download.macromedia.com/pub/labs/flashplatformruntimes/shared/air3-2_flashplayer11-2_p6_releasenotes.pdf ?

Maybe you can rewrite the Bitmap Test from http://www.craftymind.com/guimark3/ for stage3d. It would allow a nice comparison to "normal" flash and html5 canvas on android devices.

I can test with a Samsung Galaxy S II device.

[Moczan]

If we are benchmarking 2D rendering, why not use Bunny Benchmark? It has batch rendering with alpha+rotation+scale and distorted background. Also, it would be easier to compare it to other platforms like haxeNME for which such benchmarks already exist.

[st33d]

Looking up the Bunny Benchmark he seems to be doing other stuff as well as the rendering. I'm not sure depth sorting or offscreen rendering is relevant.

Pure rendering numbers are what matter because that's the limit you step back from.

[st33d]

Just knocked up some basic tests today for my control group data. MovieClips and blitting stats. I used 32x32 solid colour bitmaps in MovieClips, as well as 32x32 24bit alpha pngs in MovieClips. For the blitting tests I renderered the MovieClip content to bitmapDatas.

The code simply adds items to render until the framerate drops, then removes them, then starts adding when the framerate bounces back up.

method	first brake	brake	release
Static MovieClip	7860	12394	12319
Static Alpha MovieClip	1700	3473	3389
Moving MovieClip	4140	4165	3773
Moving Alpha MovieClip	1120	1428	1009
Moving Rotating MovieClip	2360	2634	2162
Moving Rotating Alpha MovieClip	1000	1041	661
Static Blitting	12000	12147	11596
Static Alpha Blitting	10780	10768	10294
Moving Blitting	10780	10848	10331
Moving Alpha Blitting	10180	9987	9987

Apologies for TIGSource's tables.

I know that some might complain that the rubberbanding might not be relevant, but with the rubberbanding, Flash, on the first test of static MovieClips supported over 19,000 MovieClips on stage at 30fps, and would then bounce back and forth, inching higher.

MovieClips it seems are actually very well optimised, and that optimisation scales for static MovieClips.

Blitting on the other hand always gives a consistent amount, around 10,000 elements.

This is the code for the first test:

Code:

package  {
	import com.nitrome.util.FPS;
	import flash.display.MovieClip;
	import flash.display.Sprite;
	import flash.events.Event;
	import flash.filters.GlowFilter;
	import flash.text.TextField;
	
	/**
	 * Rubberbanded benchmark - tries to maintain 30fps
	 * 
	 * result:
		Static MovieClip Test
		fps: 31
		MovieClips: 13776
		first brake: 7860
		brake: 12394
		release: 12319 
	 * 
	 * @author Aaron Steed, nitrome.com
	 */
	public class Game extends Sprite {
		
		public var _root:MovieClip;
		public var status:TextField;
		public var speed:int;
		public var brake:Boolean;
		
		public var firstBrake:int;
		public var currentBrake:int;
		public var currentRelease:int;
		
		// temps
		private var mc:MovieClip;
		private var i:int;
		
		public static const WIDTH:Number = 550;
		public static const HEIGHT:Number = 550;
		public static const FPS_LIMIT:int = 30;
		public static const DEFAULT_SPEED:int = 20;
		
		// library assets with a 32x32 solid colour png in each
		public static const MCS:Array = [NoAlphaMC1, NoAlphaMC2, NoAlphaMC3, NoAlphaMC4];
		
		public function Game(root:MovieClip) {
			_root = root;
			FPS.start();
			speed = DEFAULT_SPEED;
			status = new TextField();
			status.width = 200;
			status.height = 200;
			status.text = "0 fps";
			status.selectable = true;
			status.filters = [new GlowFilter(0xFFFFFF, 1, 4, 4, 2000)];
			_root.addChild(this);
			_root.addChild(status);
			addEventListener(Event.ENTER_FRAME, main);
		}
		
		public function main(e:Event):void{
			
			// if fps is stable at 30
			if(FPS.value >= FPS_LIMIT){
				if(!brake){
					currentRelease = numChildren;
					brake = true;
				}
				for(i = 0; i < speed; i++){
					mc = new MCS[(Math.random() * MCS.length) >> 0];
					mc.x = Math.random() * (WIDTH - mc.width);
					mc.y = Math.random() * (HEIGHT - mc.height);
					addChild(mc);
				}
				
				
			} else if(numChildren){
				if(brake && speed > 1){
					if(firstBrake == 0) firstBrake = numChildren;
					currentBrake = numChildren;
					brake = false;
					speed--;
				}
				removeChildAt(numChildren-1);
			}
			
			status.text = "Static MovieClip Test\nfps: " + FPS.value + "\nMovieClips: " + numChildren + "\nfirst brake: " + firstBrake + "\nbrake: " + currentBrake + "\nrelease: " + currentRelease;
		}
		
	}

}

I'll try to upload some swfs and more source code. Just want to try out bitmapData.draw benchmarks before I move on to benchtesting Starling and other stuff.

[Fallsburg]

What do you mean by alpha blitting?  Is that copyPixels with an alphaBitmapData?

[Richard Kain]

I'm a little curious as to why you're using MovieClips. I stopped using that class years ago, and never looked back. I either use Shape, Sprite, or Bitmap as the root classes for display objects in my Flash projects. (depending on how interactive I need them to be) Unless you were trying to import animations and graphics that had been constructed with the Flash IDE, I can't think of any reason to touch the MovieClip class.

[st33d]

Our teams are organised of one dedicated pixel artist/flash animator and one programmer. The MovieClips are used by the artists to fashion the animations, gui design and other artwork to their precise aesthetic. Bear in mind I work for Nitrome, we get criticism for everything but the art, we know what we're doing there.

I generally rasterise most of the MovieClips in a setup function, but for rotation or some animators and their addiction to filters and nesting, we have to use MovieClips to get the projects done in a profitable turn over.

What do you mean by alpha blitting?  Is that copyPixels with an alphaBitmapData?

Sort of. I created BitmapDatas with low alpha values to try to cause as much computation as possible. It seems to have only a marginal effect on blitting and a massive one on the MovieClips. I'm not using an alpha mask though.

I think I'll try and set up a new repo at Nitrome's github and upload all of the test cases. I've put them as separate projects.

[st33d]

So here's the graph for the control group:



Interesting to note on the BitmapData.draw that scaling has little impact but rotating does.

Going to press on with Stage3D tests now.

[Fallsburg]

So, I'd propose a couple of other things:
1) colorTransform for alpha.  I don't think that precomputed rotations or alphas should count since it's sort of against the spirit of the test. 

2)Nevermind, I see you have the draw tests in there.

[st33d]

Okay. After some discussion over whether I had screwed up the code, some stats for Starling:



This is just my work machine, which isn't great, but still. The figures are pretty bad for cross-platform development.

Here's the code from one of the Starling tests in case someone can see where I messed up, if at all:

Code:

package  {
	import com.nitrome.util.FPS;
	import flash.display.BitmapData;
	import flash.events.Event;
	import flash.filters.GlowFilter;
	import flash.text.TextField;
	import starling.display.DisplayObject;
	import starling.display.Sprite;
	import starling.display.MovieClip;
	import starling.textures.Texture;
	import starling.textures.TextureAtlas;
	
	/**
	 * Rubberbanded benchmark - tries to maintain 30fps
	 * 
	 * result:
		DirectX9:
		Moving Starling MovieClip Test
		fps: 31
		MovieClips: 1644
		first brake: 1720
		brake: 1781
		release: 1320
		
	 * 
	 * @author Aaron Steed, nitrome.com
	 */
	public class Game extends Sprite {
		
		public static var _root:*;
		public var status:TextField;
		public var speed:int;
		public var brake:Boolean;
		
		public var firstBrake:int;
		public var currentBrake:int;
		public var currentRelease:int;
		
		public var sprites:Vector.<Sprite>;
		public var textures:Vector.<Texture>;
		public var texture:Texture;
		public var atlas:TextureAtlas;
		
		
		// temps
		private var sprite:Sprite;
		private var mc:MovieClip;
		private var displayObject:DisplayObject;
		private var i:int;
		private var length:int;
		
		public static const WIDTH:Number = 550;
		public static const HEIGHT:Number = 550;
		public static const FPS_LIMIT:int = 30;
		public static const DEFAULT_SPEED:int = 20;
		
		public function Game() {
			FPS.start();
			speed = DEFAULT_SPEED;
			sprites = new Vector.<Sprite>();
			
			texture = Texture.fromBitmapData(new AtlasBD(1, 1), false);
			var xml:XML = <TextureAtlas>
				<SubTexture name='_1' x='0'  y='0' width='32' height='32'/>
				<SubTexture name='_2' x='32' y='0' width='32' height='32'/>
				<SubTexture name='_3' x='64' y='0' width='32' height='32'/>
				<SubTexture name='_4' x='96' y='0' width='32' height='32'/>
				<SubTexture name='alpha_1' x='128'  y='0' width='32' height='32'/>
				<SubTexture name='alpha_2' x='160' y='0' width='32' height='32'/>
				<SubTexture name='alpha_3' x='192' y='0' width='32' height='32'/>
				<SubTexture name='alpha_4' x='224' y='0' width='32' height='32'/>
			</TextureAtlas>;
			atlas = new TextureAtlas(texture, xml);
			textures = atlas.getTextures("_");
			
			status = new TextField();
			status.width = 200;
			status.height = 200;
			status.selectable = true;
			status.filters = [new GlowFilter(0xFFFFFF, 1, 4, 4, 2000)];
			_root.addChild(status);
			
			_root.addEventListener(Event.ENTER_FRAME, main);
		}
		
		public function main(e:Event):void{
			
			// if fps is stable at 30
			if(FPS.value >= FPS_LIMIT){
				if(!brake){
					currentRelease = numChildren;
					brake = true;
				}
				for(i = 0; i < speed; i++){
					texture = textures[(Math.random() * textures.length) >> 0];
					mc = new MovieClip(Vector.<Texture>([texture]), 30);
					mc.x = Math.random() * (WIDTH - mc.width);
					mc.y = Math.random() * (HEIGHT - mc.height);
					addChild(mc);
				}
				
			} else if(numChildren){
				if(brake && speed > 1){
					if(firstBrake == 0) firstBrake = numChildren;
					currentBrake = numChildren;
					brake = false;
					speed--;
				}
				removeChildAt(numChildren - 1);
			}
			
			// move movieclips manually
			for(i = 0; i < numChildren; i++){
				displayObject = getChildAt(i);
				displayObject.x++;
				if(displayObject.x > WIDTH - displayObject.width) displayObject.x = 0;
			}
			
			status.text = "Moving Starling MovieClip Test\nfps: " + FPS.value + "\nMovieClips: " + numChildren + "\nfirst brake: " + firstBrake + "\nbrake: " + currentBrake + "\nrelease: " + currentRelease;
		}
		
	}

}

[Fallsburg]

That looks right for Starling.  Moving/Scaling/Rotating/Alpha should all be free, essentially, and your data bears that out. 

I'd suggest doing nd2d, specifically, check out the supplied stress test for numbers of sprites.  As I said earlier, the Sprite2DCloud or Sprite2DBatch in nd2d should be much faster than Starling.

[st33d]

Okay. Stats in for ND2D:



Um. Well it can compete with fancy standard MovieClips, but standard MovieClips are still ahead and blitting is still miles ahead.

Going to pick apart ND2D now for a raw approach before uploading these projects to git.

Here's code for the static test in case anyone can see if I'm going about this wrong:

Code:

package  {
	import com.nitrome.util.FPS;
	import de.nulldesign.nd2d.display.Scene2D;
	import de.nulldesign.nd2d.display.Sprite2D;
	import de.nulldesign.nd2d.display.Sprite2DBatch;
	import de.nulldesign.nd2d.display.Sprite2DCloud;
	import de.nulldesign.nd2d.display.World2D;
	import de.nulldesign.nd2d.materials.texture.SpriteSheet;
	import de.nulldesign.nd2d.materials.texture.Texture2D;
	import de.nulldesign.nd2d.materials.texture.TextureAtlas;
	import flash.display.BitmapData;
	import flash.display.DisplayObject;
	import flash.events.Event;
	import flash.filters.GlowFilter;
	import flash.geom.Rectangle;
	import flash.text.TextField;
	import flash.display3D.Context3DRenderMode;
	
	/**
	 * Rubberbanded benchmark - tries to maintain 30fps
	 * 
	 * result:
		
		Static ND2D SpriteBatch Test
		fps: 25
		Sprites: 3415
		first brake: 3480
		brake: 3416
		release: 3016
		
	 * 
	 * @author Aaron Steed, nitrome.com
	 */
	public class Game extends World2D{
		
		public static var _root:*;
		public var status:TextField;
		public var speed:int;
		public var brake:Boolean;
		
		public var firstBrake:int;
		public var currentBrake:int;
		public var currentRelease:int;
		
		public var spriteBatch:Sprite2DBatch;
		public var spriteSheet:SpriteSheet;
		
		// temps
		private var i:int;
		private var length:int;
		private var sprite:Sprite2D;
		
		public static const WIDTH:Number = 550;
		public static const HEIGHT:Number = 550;
		public static const FPS_LIMIT:int = 30;
		public static const DEFAULT_SPEED:int = 20;
		
		public function Game() {
			super(Context3DRenderMode.AUTO, 30);
		}
		
		override protected function addedToStage(event:Event):void {
			super.addedToStage(event);
			
			FPS.start();
			speed = DEFAULT_SPEED;
			
			var bd:BitmapData = new AtlasBD(1, 1);
			var scene:Scene2D = new Scene2D();
			setActiveScene(scene);
			
			
			var texture:Texture2D = Texture2D.textureFromBitmapData(bd);
			spriteSheet = new SpriteSheet(256, 32, 32, 32, 30, true);
			spriteBatch = new Sprite2DBatch(texture);
			spriteBatch.setSpriteSheet(spriteSheet);
			scene.addChild(spriteBatch);
			
			_root.addChild(this);
			
			status = new TextField();
			status.width = 200;
			status.height = 200;
			status.selectable = true;
			status.filters = [new GlowFilter(0xFFFFFF, 1, 4, 4, 2000)];
			_root.addChild(status);
			
			_root.addEventListener(Event.ENTER_FRAME, main);
			start();
		}
		
		
		public function main(e:Event):void{
			
			// if fps is stable at 30
			if(FPS.value >= FPS_LIMIT){
				if(!brake){
					currentRelease = spriteBatch.numChildren;
					brake = true;
				}
				for(i = 0; i < speed; i++){
					sprite = new Sprite2D();
					spriteBatch.addChild(sprite);
					sprite.spriteSheet.frame = Math.random() * 4;
					sprite.x = Math.random() * (WIDTH - 32);
					sprite.y = Math.random() * (HEIGHT - 32);
				}
				
			} else if(spriteBatch.numChildren){
				if(brake && speed > 1){
					if(firstBrake == 0) firstBrake = spriteBatch.numChildren;
					currentBrake = spriteBatch.numChildren;
					brake = false;
					speed--;
				}
				spriteBatch.removeChildAt(spriteBatch.numChildren - 1);
			}
			
			status.text = "Static ND2D SpriteBatch Test\nfps: " + FPS.value + "\nSprites: " + spriteBatch.numChildren + "\nfirst brake: " + firstBrake + "\nbrake: " + currentBrake + "\nrelease: " + currentRelease;
		}
		
	}

}

[Fallsburg]

I had another thought about making this closer to a realistic test.  I was thinking something along the lines of, construct a maze (or other sort of map or something) and do A* pathfinding every frame (couple of frames) or something like that.  Part of the benefit of offloading to the GPU should be that it frees up CPU time for the other portions of your game, so I'd be interested to see what the tradeoff in CPU vs GPU when other things are trying to use the CPU.

Also, I'm a bit surprised at your nd2d results. On my computer I get about 4 times as many sprites with Sprite2DBatch than I do with Starling (admittedly this was Starling 4 months ago, maybe they have tightened things up since then).  I might have to run the tests on my computer to see what my results are.

[st33d]

Why calculate A* when you could just as easily run an empty for-loop?

I've ran all these tests in the Debug Player by the way. Which is the standard playtest environment for us - it's impossible to catch bugs otherwise, suggesting otherwise is like burying your head in the sand.

Testing the static test on ND2D in the standard 11.2 player gives an extra 1000 objects.

The purpose of the tests are to find worst case scenarios. There's a lot of press saying that Stage3D outperforms other forms of rendering, and the truth seems to be - if you have an amazing new graphics card.

Where does that leave people who don't have an amazing graphics card and how do we identify them?

[Fallsburg]

I was just trying to come up with something representative of actual in game usage.  I guess you could just parameterize over number of iterations to see how it scales.

I would suggest doing these tests in both the debug and non-debug player, if only for the sake of information.

Blitting will typically always be faster than rendering (except on some corner case computer with a horrible CPU and phenomenal GPU), due to the fact that GPUs just aren't set up to handle something like sprites (i.e. they are designed to handle a small number of complex objects, not a large number of simple objects), but the GPU should pretty much always win when you try to do anything more advanced than just moving around static sprites.