Survey of Fast Compression Algorithms (Part 1)

Recently the subject of compression algorithms was raised on Twitter, and a number of new additions to the field were mentioned in the ensuing discussion. As I’m currently working on improving load times for a project, these newcomers piqued my interest and I set out to do some evaluation of them with “game-ish” assets.

I haven’t been able to test every algorithm I wanted to get to yet (new and old), and only have PC results to show so far – but am releasing the source so it can be extended as people desire. See the end of the post for a download link. Future algorithms on my list to add to the test suite are ZLib, lzham, LZMA, and a few variants there of.

UPDATE April 24, 2011: LZ4 has been open-sourced, so I’ve added it to the test suite and updated the results. Charts have also been added to make comparison easier.

The test data I chose came mainly from the DirectX SDK samples, with a few files that were kicking around my hard drive for good measure. Highly scientific! I hope to do more realistic tests soon!

Tests were performed on a low-end Core2 Duo laptop. Without further delays, here’s the numbers.

 

Summary Results:

 

Compression Ratio (click to enlarge):

Compression Time (click to enlarge):

Compression Time, Zoomed (click to enlarge):

 

Decompression Time (click to enlarge):

Decompression Time, Zoomed (click to enlarge):

 

Summary Results:

Algorithm

% (Average)

Compression Time (Total)

Decompression Time (Total)

Doboz 66.609 38.26463 0.24643
Yappy 76.616 1.83406 0.10750
Snappy 73.838 0.51189 0.20106
LZF 72.293 0.73531 0.20450
FastLZ 73.105 0.89960 0.25096
QuickLZ 73.373 0.44819 0.29023
LZ4 71.743 0.52270 0.13279

 

Compression Details:

  Doboz Yappy
File Type % S % S
BMP 76.859 0.28196 100.511 0.01840
BMP 15.771 2.04379 39.693 0.08194
DDS 92.847 2.71034 98.261 0.13022
DDS 100.000 1.15703 102.474 0.06402
DDS 30.457 0.38021 37.101 0.01224
DDS 58.066 1.63392 78.431 0.07035
DDS 77.604 0.21672 85.799 0.01651
DDS 38.272 0.32217 45.296 0.01378
DDS 75.037 2.75669 98.596 0.09562
DDS 50.520 2.97517 81.242 0.09691
DDS 38.885 0.07744 46.294 0.00358
DDS 98.550 0.04572 100.713 0.00609
DDS 46.584 7.37891 72.888 0.28009
X 17.017 0.05599 28.431 0.00297
JPG 100.001 0.15212 101.159 0.02035
PDF 100.000 0.61228 102.308 0.06216
PDF 100.000 5.26727 98.232 0.36000
PNG 100.032 0.00551 101.561 0.00038
PNG 100.018 0.00658 101.584 0.00065
MESH 30.941 0.16696 47.942 0.01962
ANIM 20.349 0.08969 23.585 0.00349
SMD 8.705 0.81705 28.595 0.02770
TXT 36.946 0.01829 54.240 0.00270
TXT 36.000 0.00952 49.461 0.00078
HDR 85.190 3.04351 97.053 0.11724
WAV 100.000 1.34706 95.849 0.09014
WAV 100.000 0.98768 99.833 0.07076
WAV 97.022 2.64150 102.726 0.10698
WAV 100.000 0.83595 102.019 0.06339
  Snappy LZF
File Type % S % S
BMP 98.494 0.00494 86.235 0.00886
BMP 34.023 0.02537 42.279 0.01739
DDS 97.980 0.04788 95.804 0.06335
DDS 99.956 0.00500 100.229 0.02882
DDS 36.008 0.00356 33.066 0.00441
DDS 76.513 0.04170 70.303 0.02679
DDS 86.051 0.00662 82.233 0.00726
DDS 44.553 0.00433 42.180 0.00441
DDS 96.407 0.03740 86.483 0.04520
DDS 76.097 0.05383 68.093 0.03548
DDS 45.953 0.00131 43.822 0.00149
DDS 98.711 0.00064 97.998 0.00281
DDS 68.686 0.15584 62.146 0.09891
X 25.919 0.00096 26.458 0.00067
JPG 97.265 0.00130 100.833 0.00837
PDF 98.477 0.00447 101.311 0.02401
PDF 95.421 0.03683 97.329 0.14390
PNG 99.440 0.00006 101.189 0.00036
PNG 99.958 0.00005 100.406 0.00037
MESH 41.180 0.00460 40.191 0.00450
ANIM 22.804 0.00068 21.442 0.00069
SMD 19.854 0.00573 15.934 0.00462
TXT 51.544 0.00083 49.048 0.00067
TXT 45.939 0.00033 45.320 0.00032
HDR 94.695 0.03526 91.182 0.05647
WAV 93.265 0.01055 94.968 0.04045
WAV 97.076 0.00619 99.201 0.02845
WAV 99.861 0.00718 99.541 0.04940
WAV 99.176 0.00445 101.271 0.02595
  FastLZ QuickLZ
File Type % S % S
BMP 90.305 0.01170 91.482 0.00714
BMP 41.562 0.02124 49.502 0.01913
DDS 97.465 0.08657 100.000 0.02873
DDS 101.011 0.03795 100.000 0.01411
DDS 33.720 0.00467 33.060 0.00449
DDS 74.448 0.03656 69.147 0.02293
DDS 85.192 0.01019 87.021 0.00553
DDS 43.315 0.00570 42.407 0.00355
DDS 89.005 0.05589 86.723 0.03342
DDS 70.850 0.04215 69.727 0.02931
DDS 45.153 0.00162 43.871 0.00100
DDS 98.450 0.00313 100.003 0.00219
DDS 67.313 0.11819 64.563 0.07868
X 25.687 0.00079 24.202 0.00057
JPG 99.935 0.00988 100.001 0.00424
PDF 101.166 0.02915 100.000 0.01258
PDF 97.352 0.17365 100.000 0.07601
PNG 101.303 0.00018 100.057 0.00008
PNG 100.720 0.00032 100.032 0.00013
MESH 36.532 0.00609 37.332 0.00426
ANIM 20.880 0.00087 21.235 0.00069
SMD 14.640 0.00526 14.675 0.00456
TXT 49.004 0.00062 47.658 0.00044
TXT 45.977 0.00024 45.105 0.00019
HDR 93.522 0.06468 100.000 0.02546
WAV 95.033 0.04300 100.000 0.01853
WAV 99.288 0.03458 100.000 0.01526
WAV 99.819 0.05917 100.000 0.02266
WAV 101.391 0.03263 100.000 0.01352
  LZ4
File Type % S
BMP 92.720 0.01104
BMP 36.492 0.01010
DDS 93.103 0.07531
DDS 98.942 0.02954
DDS 31.981 0.00348
DDS 69.793 0.02250
DDS 79.072 0.00957
DDS 40.509 0.00522
DDS 86.819 0.04902
DDS 73.269 0.02903
DDS 42.889 0.00134
DDS 97.659 0.00189
DDS 66.108 0.06883
X 26.743 0.00113
JPG 97.413 0.00170
PDF 98.492 0.00430
PDF 95.267 0.03972
PNG 99.276 0.00040
PNG 99.086 0.00063
MESH 38.591 0.00512
ANIM 20.849 0.00152
SMD 15.971 0.00518
TXT 51.581 0.00079
TXT 49.976 0.00052
HDR 89.331 0.06100
WAV 93.001 0.01503
WAV 97.113 0.00956
WAV 99.146 0.05368
WAV 99.346 0.00570

 

Decompression Details:

  Doboz Yappy Snappy
File Type S S S
BMP 0.00540 0.00067 0.00114
BMP 0.00757 0.00556 0.01270
DDS 0.02847 0.00675 0.01048
DDS 0.00452 0.00294 0.00359
DDS 0.00114 0.00053 0.00110
DDS 0.01542 0.00680 0.01513
DDS 0.00218 0.00075 0.00156
DDS 0.00132 0.00055 0.00124
DDS 0.02774 0.00535 0.01060
DDS 0.01797 0.00713 0.01974
DDS 0.00040 0.00017 0.00040
DDS 0.00074 0.00029 0.00021
DDS 0.04511 0.02750 0.05855
X 0.00023 0.00017 0.00034
JPG 0.00059 0.00054 0.00118
PDF 0.00260 0.00249 0.00308
PDF 0.02046 0.01731 0.02017
PNG 0.00001 0.00001 0.00001
PNG 0.00001 0.00001 0.00001
MESH 0.00197 0.00085 0.00284
ANIM 0.00019 0.00013 0.00129
SMD 0.00214 0.00180 0.00479
TXT 0.00020 0.00012 0.00029
TXT 0.00009 0.00005 0.00012
HDR 0.03745 0.00647 0.01246
WAV 0.00334 0.00302 0.00409
WAV 0.00269 0.00242 0.00431
WAV 0.02631 0.00529 0.00710
WAV 0.00233 0.00221 0.00455
  LZF FastLZ QuickLZ
File Type S S S
BMP 0.00276 0.00416 0.00679
BMP 0.01091 0.01349 0.01799
DDS 0.01478 0.01770 0.00538
DDS 0.00542 0.00613 0.00290
DDS 0.00107 0.00126 0.00243
DDS 0.01122 0.01429 0.02223
DDS 0.00187 0.00234 0.00536
DDS 0.00129 0.00157 0.00305
DDS 0.01678 0.02031 0.03722
DDS 0.01649 0.01970 0.03277
DDS 0.00041 0.00047 0.00082
DDS 0.00054 0.00067 0.00011
DDS 0.04369 0.05458 0.08630
X 0.00036 0.00040 0.00047
JPG 0.00092 0.00119 0.00057
PDF 0.00403 0.00379 0.00221
PDF 0.02469 0.02823 0.01733
PNG 0.00002 0.00002 0.00001
PNG 0.00003 0.00004 0.00001
MESH 0.00192 0.00213 0.00528
ANIM 0.00019 0.00021 0.00049
SMD 0.00292 0.00324 0.00441
TXT 0.00025 0.00134 0.00041
TXT 0.00032 0.00013 0.00017
HDR 0.01578 0.01920 0.00687
WAV 0.00485 0.00668 0.02031
WAV 0.00393 0.00441 0.00260
WAV 0.01349 0.02018 0.00550
WAV 0.00467 0.00572 0.00242
  LZ4
File Type S
BMP 0.00145
BMP 0.00668
DDS 0.00939
DDS 0.00319
DDS 0.00075
DDS 0.00831
DDS 0.00107
DDS 0.00086
DDS 0.01101
DDS 0.01046
DDS 0.00026
DDS 0.00021
DDS 0.03002
X 0.00026
JPG 0.00086
PDF 0.00276
PDF 0.01641
PNG 0.00001
PNG 0.00002
MESH 0.00128
ANIM 0.00013
SMD 0.00228
TXT 0.00017
TXT 0.00009
HDR 0.01020
WAV 0.00393
WAV 0.00239
WAV 0.00535
WAV 0.00250

 

Current source can be grabbed here (mostly platform-agnostic test suite library with a C# GUI and C++/CLI for interop):
http://technofumbles.com/files/CompressionTest.zip

Apr 22nd, 2011 | Filed under AltDevBlogADay, GameDev

Tips for High Performance Art

While performance is often thought of as a programmers problem, the truth is that even some simple changes to how art is authored can have a drastic affect on a games performance. Unfortunately programmers suck at divulging much of this information – something I hope to remedy somewhat in this post. As with anything performance related, this isn’t an absolute list of rules to live by – some will depend on the architecture of the engine you’re working with, others on the particular scene, and yet others on the artistic look and style of the game. It’s also by no means comprehensive; I think this will likely spill over into another blog post. It’s best to discuss with your rendering team how (and if) each of these impact your game.

Pixel Quads and Pixie Dust

A few days ago an artist asked me for information on specular behaviour for materials, to which I sent him a link to the post Everything is Shiny over on the excellent Filmic Games blog. While browsing the site, he came across the post Where are my sub-pixel triangles???, which discusses performance issues arising from small triangles in meshes. Amazed that he’d never heard of such issues he came back and asked me if it really was true – and it is. It’s just never really discussed beyond the programming tracks at GDC, Gamefest, and so on. By the time it gets to artists it probably has been reduced to "make sure there are LODs", with not further details on what this means or why they need to exist. After all, LODs just reduce the vertex count – that couldn’t possibly affect the pixel shading performance! Right? It turns out that this is absolutely incorrect, and LODs have in fact a dramatic affect on pixel shading performance.

What. The. Hell. Yeah, it sounds odd. Your LOD is half the vertex count of the full-resolution mesh but still takes up exactly the same number of pixels on screen as the full mesh at the same distance from the camera, however the pixel shading performance of the full mesh is much lower than that of the LOD?! This is because there is a trade-off in the hardware between the size of triangles and pixel shader performance. The trade-off varies between hardware, and in some cases can be very extreme, but it never doesn’t exist.

Being a hardware trade-off this does mean that I have to explain a bit about how the hardware works when a triangle is rendered.

The process of rendering a mesh goes something like this:

Vertex processing (vertex shader) -> MAGIC –> Pixel processing (pixel shader)

What is the MAGIC step? That depends on the platform, but the general formula is that each triangle is broken into 2×2 blocks of pixels called quads, and then some (potentially large) number of quads are run through the pixel shader at once. As the X360 has a lot of great documentation on this publicly available from Gamefest, let’s take that as an example (the PS3 isn’t all that different anyway).

Putting this together, we get:

Vertex processing –> Triangles –> Quads -> 16-Quads –> Pixel processing

This might start to ring alarm bells as to why small triangles are expensive. To take the worst-case that you have a single 1 pixel triangle, for that single pixel you might as well be executing the pixel shader 64 (2*2*16) times! If you have 64 single-pixel triangles so there are enough quads generated to fill a full pixel vector, you will still at best have 25% pixel shading efficiency due to the 2×2 quads. To put this another way, the lighting, texture filtering, and all-round shiny-awesomeness of the pixels could be FOUR times as complex if we avoid these cases. There’s nothing that anybody but the artist creating the mesh can do about this either (other than a programmer running some ugly automatic decimation algorithm on the mesh in the pipeline, which I’ve never seen any artist happy about – for good reason). You won’t be able to make a perfect mesh, but it’s important to make a good one.

This can be distilled down to a simple ratio for easy comparison: vertex to pixel quad density. Ask a rendering programmer to show you how to figure this out for your art, or see the presentations listed in the References section at the end of this post.

A few other random stats pulled from various Gamefest presentations:

  • Triangles that cover 2×2 pixels waste 60% or more of the pixel shading performance
  • Going from 2×2 pixel triangles to 4×4 can increase the pixel shading performance by 33%

Overdraw Overload

The cost of overdraw is proportional to the speed at which memory can be written to of the platform and the size of the screen. It may come as a surprise to many, but the PS2 actually performed better in this area than the X360 and PS3 (or even most modern PC graphics cards). As such, overdraw is actually quite significantly more expensive for this generation of consoles than it was for the previous (at least, on the PS2). We are saved somewhat by more flexible hardware with optimizations to fight the cost of overdraw, but those unfortunately don’t help when the overdraw consists of alpha-blended geometry (particles are obviously the worst for this).

So uh… that’s about it – overdraw is still a terrible problem for alpha-blended geometry, so try to reduce the number of layers of overdraw as much as possible.

Light, Shadow and Their Dark Interactions

Low-angle light directions are the enemy of efficient shadows, as they require far more objects to be classified as casting shadows for a frame than otherwise. The easiest way to look at this is that the length of the shadow that an object casts directly relates to how expensive it will be to be shadowed. This isn’t about the actual cost of calculating the shadow for the particular object, but about the cost to include it in the shadow calculations because the longer a shadow than an object can cast, the further outside of the current cameras view it must still be evaluated since it could be casting a shadow over something that is visible to the camera.

It doesn’t help that these low-angle light directions make for the most interesting lighting environments of course, so a compromise will probably have to be reached between the art and programming teams.

Death by Draw Calls

Everyone hates unnecessary bookkeeping; it’s red tape in the way of getting things done, and pulls you away from what you really want to be doing. Generally speaking, if you have different materials (shaders, textures, lighting values, etc.) applied to different parts of a mesh, it will have to be split up by the engine and each piece rendered individually. When you have multiple meshes on multiple objects split into multiple parts, you’re cutting down the amount of useful work that the GPU and engine can do as it’s having to do a lot of bookkeeping instead.

Try to use texture pages wherever possible, UV some parts of a mesh to a black part of the specular texture rather than a separate material with no specular, and weld together any meshes that can be.

Summary

This is about half of the items that I wanted to cover in this post, so it’ll definitely be covered further in my next blog post. But for now that should be plenty to chew on, and certainly covers a lot of the bigger issues affecting performance of art.

References

Technical Art Reviews: Getting the Most Out of your Content, Cameron Egbert, Gamefest 2010
PIX for Artists, Alan Murphy, Gamefest 2010
Xbox Textures – Formats, Conversion, Fetching, and Performance, David Cook and Jason Gould, Gamefest 2008
Xbox 360 Shaders and Performance: How Not to Upset the GPU, Cody Pritchard, Gamefest 2010
Building an Über-Fast Crowd Renderer for Your Next Xbox 360 Engine, Tomas Vykruta, Gamefest 2010
Past Gamefest Presentations

Feb 21st, 2011 | Filed under AltDevBlogADay, GameDev

The Importance of Personal Projects

Personal projects are an often overlooked part of a professional career – be it games, embedded, or even web development. There’s only so much you will be exposed to in an office environment, where you can lean on others for areas that aren’t your specialty. But when it comes to personal projects, you have to be a jack of all trades.

This is the reason I ask about hobbies and personal projects of those who I interview and those for whom I interview with (interviews go both ways remember!). It’s not something I ask because I want to know if you’ll be my buddy, even though that is very important factor in selecting coworkers in my opinion, but because it is one of the most clear indicators of a curious and active mind – and that trait is very important. An engineering team with curious and active minds is the cornerstone of a solid development process. They are the ones who will be the most forward-looking, identifying the potential pitfalls ahead and working to avoid them now, because they’re thinking. They will be versatile, because they’re used to it. They will actively improve the codebase, systems and procedures, because they want to work somewhere that they enjoy what they work with.
Great work is not achieved by punching a clock, and someone who is curious will work on the problem because they want to solve it. This isn’t to say that I like curious coworkers because they do lots of overtime – my opinion on overtime is not positive at all – quite the opposite in fact. I don’t really care the hours you do if you’re doing a great job in that time and not slacking off. The unfortunate reality of the games industry is that you will be doing overtime, and that overtime will make up for any short days taken. If you’re making no progress on a problem I’d much rather see my team take off early and relax than sit around bored and frustrated – then when the solution finally comes you’ll be fresh and happy to work hard on doing a great job rather than burnt out and doing just a good job.

Of course that isn’t to say that if you don’t have any personal projects on the go that you don’t have a curious and active mind, only that it’s an a clear indicator in an interview; I know plenty of great engineers who don’t fit this profile, and there’s nothing wrong with that. There will be things that you’ll miss out on though, because often there isn’t time in the schedule to bring someone up to speed on a new area, meaning that if you don’t already know it, you won’t get the opportunity to learn. This isn’t for any malicious reason; it’s simply the reality of schedules.

There’s no reason that a personal project has to be directly related to your daily work. If it was, that’d be little more than homework. Not much fun there if that’s all you do. For instance, my personal projects are usually a mix of embedded (electronics and low-level programming), Android (mobile programming and OpenGL), and the occasional work-related R&D. What counts is that you’re pushing yourself to learn, and having fun doing it.

Feb 5th, 2011 | Filed under AltDevBlogADay, GameDev

Intuition for Gamma Correct Rendering

I want to start out by apologising for a lack of pretty graphics in here, which is a little odd for a post about visual quality. It’s a simple answer as to why this is though – I’m currently typing this on my laptop, watching as the Windows 7 system recovery progress bar loops on the screen of my main PC (uh… yeah, it’s behaving a little odd. But that’s another topic entirely).

Ok, now that you know you’re in for a lot of monotonous text, let’s get on with it!

Gamma correct rendering may sound like a simple enough concept at first, but to do it correctly can be very challenging – especially once you throw hardware variations into the mix. Possibly worse yet is that it is something you must keep in mind throughout development, and educate your teammates about. Or you ignore the issue completely and live with the consequences, but it will come around to bite you down the road. Repeatedly.

First some definitions:

  • <Gamma/sRGB/Linear> space
    The curve of raw data values to represented values for the data that you’re working with (could by bytes, floats, or rgb triple). Linear is easy as it’s an exact match. Gamma and sRGB on the other hand are not identical, and instead define a curve that gives more raw data values to the lower ranges of the represented values compared to the higher ranges. For the purpose of this post, we’ll call gamma and sRGB the same thing (though they aren’t necessarily this way, as sRGB refers to a very specific curve, while gamma can be anything).
  • To gamma a texture/value
    To convert from linear space to gamma space. This will result in raw linear values (the same as their represented values) of 0.2 going to raw gamma values of ~0.5 (but still representing 0.2). The exact values depend on the curve of the gamma space that you convert to of course.
  • To degamma a texture/value
    The opposite of the above; to convert from gamma space to linear space by applying the inverse of the gamma curve. So, with the above example you would get back the original linear raw value of 0.2 from the gamma value ~0.5.

So what does it really take to be gamma correct? There are 3 primary areas of concern: the pipeline, the shader, and the render target.

The Pipeline and Tools

  • Some of the source data that you’re given will be in gamma space, and other source data will be in linear space. The pipeline has to know what it has, and what it should do with it. Easier said than done, as this requires meta data to be present – either artist-set or automatically-set based on usage. Having artists specify what everything should be interpreted as is obviously the easiest choice when faced with having to shoehorn gamma correct behaviour into an existing pipeline without the backend architecture to support it, but does carry with it the consequence that user-errors will be abundant.
  • If you’re going to perform any processing on textures in the pipeline (resizing, mipmap generation, blending the edges of cubemap faces, etc), the operations must be done in linear space. The gotcha with this is that you must operate with as much precision as possible throughout this process to avoid issues with quantization due to the conversion to linear space and back again. This usually means converting to a floating point texture immediately, and only converting back as the very last step. Yes, and you’ll probably have issues achieving this with some of the external libraries you use. So go and modify them too (and diverge from what’s in SVN, making taking updates all that much harder). Fun stuff.
  • The final conversion to gamma space after you’ve done your processing may also bite you due to hardware variations. If you’re lucky you’ll only be targeting platforms that have proper support for sRGB – but many are not so lucky and as such will be in for a world of fiddly pain thanks to . But at least it’s documented now, which has only happened in the last couple of years. Extremely fun stuff.

The Shader

  • Everything you do in a shader should be in linear space. Simple.
  • There are states that you can set on the various platforms to automatically convert textures when sampled from gamma space to linear space, but these states do live in different places for different platforms.

The Render Target

  • The joys of hardware variations will strike you severely here, and throw a spanner (wrench for those of North American heritage) in the works.
  • Frame buffers are usually stored in gamma space and you output linear space values from the pixel shader. Thus, blending the output of the pixel shader with the frame buffer can be done in linear space (correct) or gamma space (incorrect) depending on the platform. DX9 and the PS3 will do it incorrectly, but DX10+ and the X360 will do it correctly.

Here’s a spoiler: not everything should be gamma corrected at every step. But what does that mean? Why is that? Well, that’s exactly what this post is about!

The simplest rule for whether a texture should be treated in the pipeline as linear or not is if it was authored by an artist painting using values that give the look they want on screen (“I want this blue for the sky”), it should be treated as being in gamma space. Everything else should be linear. Except when it shouldn’t be. Crap.

Oh, and that means that vertex colours should also be treated as being in gamma space. Except when they shouldn’t be. Double crap.

What are these exceptions? Take a lightmap as an example; according to the above rule, it should be in linear space – and it should. However, it’s not uncommon to have very dark lightmaps and when in linear space this will result in excessive banding in dark areas. If the same lightmap was stored in gamma space and converted to linear space only when it was sampled in the shader, you would have far more precision (and thus less banding) in the dark areas. The trade-off here is reduced precision in the light areas, but generally that’s less noticeable thanks to human vision being more sensitive to variations in darks than lights (undoubtedly to see creatures with big fangs lurking in the shadows). A trade-off is a trade-off though, and it’s not always what you want. Crytek for instance uses a metric of something like if at least 15-20% of an image has values of less than 96, an image will be stored in gamma space, and otherwise it’ll be stored in linear space – but that’s based on little more than what works for them.

There are still cases when you don’t want to do this though, such as for normal maps, as the value range has absolutely nothing to do with brightness.

What about other things? If an artist picks a tint colour for a material, it has been selected in gamma space and thus should be converted to linear space. Same goes for vertex colours that are used similarly. Fog colours too. Oh, and colours a player picks for their character’s clothes. Seeing a pattern here? There’s a lot more to being gamma correct than just textures, and it impacts a lot of people on the team who probably don’t even know what gamma correction is.

That’s why it’s a difficult area to get right.

I’m hopeful that the next generation of consoles will address the remaining issues and allow us to be entirely gamma correct across all platforms, consistently.

Because that’s why it’s currently an impossible area to get right.

For more information (and pretty pictures), I highly recommend taking a look at the presentation Post Processing in the Orange Box from Alex Vlachos at Valve.

Jan 21st, 2011 | Filed under AltDevBlogADay, GameDev
Tags:

Moved to WordPress from Blogger

A quick heads up that I’ve moved this blog from Blogger to WordPress. For the most part this change should be quite seamless to the readers (other than a different look), however please bare with me while I wait for the search engines to update as they’re currently pointing at old files. Until then, there’ll be some 404′s if you get here via Google/Yahoo/Live/etc searches.

I’ve redirected the old RSS feed URLs to their new locations, so hopefully no feed readers will be broken by this change. If there are a lot of 404′s coming from bookmarks and links to the old files, I’ll add redirects for those too.

If you notice anything broken or weird about the site however, please leave a comment and I’ll take a look!

Oct 16th, 2008 | Filed under Miscellaneous

The PCBs have arrived!

After 6 weeks, the PCBs I ordered from BatchPCB have arrived – which was disappointingly slow even for BatchPCB. Their site gives a “worst-case” (US) delivery time of 18 days I believe, however the boards weren’t even posted until 21 days, and took that long again to arrive in my mailbox here in Canada. I expected an additional 1-1.5 weeks on top of the US delivery time as Canada Post is pretty terrible, but 3 weeks for a small envelope? Ugh.

As I mentioned in my previous post, I ordered 3 copies of the same board so I’d have room to screw up a few times without having to start over with another PCB order – and that was a good thing, as one of the three boards had several manufacturing defects. The most noticeable of the defects is a large area on the bottom layer that did not have any soldermask applied to it, and thus was coated in solder like any other pad would be. Thankfully the defect only touches a components pad that is also connected to the ground plane (where the defect is) and is at the edge of the board, so it shouldn’t require scrapping the board. The other defect was solder mask over some of the pads, which required scraping off with a scalpel.

While neither of these defects are fatal for the board, they do show a severe lack of quality control. Combined with the long delivery time, I think it’s probably going to be enough for me to cut my losses and choose a more local PCB manufacturer next time, or at the very least try somewhere else. BatchPCB is cheaper (well, no actually it isn’t), but no where near cheap enough to put up with such low quality. Depending on how cheap I’m feeling at the time, and how likely the PCB is to be working, I think I’ll either go for AP Circuits as they’re in Alberta, the next province over (http://www.apcircuits.com), or OurPCB, which is another Chinese PCB manufacturer with awesome capabilities for their prototype PCBs, like 5mil traces/space (http://www.ourpcb.com).

The next step is to order some parts that I’m missing from DigiKey, as I’d been putting it off until the boards arrived. Not that I meant to do that – I simply forgot to place the order. Of course, now a few of the parts that I need are out of stock for a few more weeks. :( I might place the order anyway and include some nearly-equivalent parts to use in the mean time, and then place another order when the correct parts are back in stock and rework the board when those arrive.

Feb 20th, 2008 | Filed under Printed Circuit Boards

PCB Manufacturing

Up until now I have been making my PCBs at home (which you can read about in my previous posts), with varying levels of success. The issues mainly arose due to my heavy use of tiny surface-mount devices (SMDs), and obsession with using the smallest board possible for a design. Whenever I’m laying out a SMD board, I can’t help but think that doing so requires the designer to be at least somewhat obsessive/compulsive due to the high level of concentration required for extended periods of time. If you can shut out the rest of the world and get lost in the process, it’s easy.

Anyway, I can’t see myself going away from the tiny-SMDs, tiny-PCBs mentality as anything else feels like I’ve let myself down whether due to being a waste of material/space/effort or for whatever other reason. This leaves me with two issues that have been causing me a lot of grief in my projects – transferring a design with 0.2mm (slightly less than 8mils, or 8/1000″) traces and trace spacing onto PCBs, and soldering components with equally tiny pins/pads without any soldermask on the PCBs. The first issue hasn’t been all that bad to deal with as it just takes more time and effort to get a good transfer, but the second issue has caused far too many boards to be scrapped. The worst part is that they get scrapped after I’ve already invested hours in them through the artwork transfer, etching, drilling, inserting vias, and soldering components.

That brings me to the subject of this post – I’ve finally decided to give a commercial PCB manufacturer a go with my latest (and greatest, of course) board. I don’t want to give anything away of the designs purpose yet, so I’ll just discuss its components in a non-specific way for now.

The board consists of:
  • 44 pad QFN (7x7mm)
  • 36 pad QFN (10x10mm)
  • 28 pad QFN (8x8mm)
  • 28 pad QFN (5x5mm)
  • 8 other smaller SMD ICs (most with no leads)
  • Around 40-45 0402 capacitors
  • Around 30 0402 resistors
  • Around 10 0603 and 0805 capacitors
  • A few dozen miscellaneous SMDs like ICs, connectors, etc
  • The PCB is 85x60mm, double sided

For a nice picture of the sizes of some of these components, check out Curious Inventors guide to Surface Mount Soldering. That little speck near the tip of his little finger is an 0402 component – and I have 70-80 of those to solder!

I decided to do a bit of shopping around as this was going to be my first attempt at having a PCB commercially manufactured – not so much to find the best price, but to find one that I had a way to verify that my design was acceptable and not be rejected in a few days time. This lead me to try two services, Advanced Circuits thanks to their Free DFM site, and BatchPCB (SparkFun Electronics) also thanks to their DFM bot.

Advanced Circuits

I was quite impressed by the quality of Advanced Circuits FreeDFM.com service since it handled my board in my first go at running it through the verifier, and the errors (on my PCB) it provided me with were understandable and gave me everything I needed to fix them. I’m not sure if I did something wrong, but I just couldn’t seem to get a good price from FreeDFMs automatic quote for my design unfortunately, so I was scared off by the prospect of paying a couple hundred dollars for a few boards of a design that I have no way of knowing would work. I’ll look at them again should I need to produce at least 10 boards, but it’ll simply not be worth it otherwise.

BatchPCB

BatchPCBs verifier script is unfortunately more crude than Advanced Circuits, and many times it returned PHP errors while trying to just upload my design. Once that was taken care of, I was confronted with a lovely error of “aperture 36 too small: 0.0056″ (or similar). Now seriously, what are they thinking? You don’t know what aperture 36 is – it’s something the PCB program makes up for submitting the files to the PCB manufacturer. The only way you can know what it’s talking about is to manually open the files, find a line similar to “%ADD36C,0.0056*%”, change the size to something huge so it’s recognisable on the board, view it in a program like ViewMate, and narrow down the problem from there. This particular error turned out to be due to a library part I was using being designed with rounded SMD pads (rather than the standard square pads) – modifying the library to use square pads fixed the problem right up.
After that was out of the way, BatchPCB did provide me with an additional error that FreeDFM did not tell me about, but I think that may have been due to FreeDFM having slightly less restrictive requirements for board layout. The final price from BatchPCB? $20 per board + $16 shipping (I ordered 3 boards, so I have a few spare just in case).

So there we go, it did take nearly a day of fiddling and Googling, but I have my first PCB order under way. As BatchPCB works as an aggregator for a Chinese PCB manufacturer, I’m not expecting to see the boards for another month due to shipping times – but maybe I’ll get a pleasant surprise.

Updates on recent posts

It’s been a few weeks since my last post, but rest assured that I’ve been quite busy working on more projects!

Homemade PCB

The home-made PCB in the previous post turned into a bit of a disaster when I tried to take the toner off the traces with a Scotch-Brite pad. It turns out that brute force and abrasive materials aren’t the best way to remove toner from 0.2mm traces. Who’d have thought? ;) So I’ve etched the board again, and took the opportunity to revise the design slightly while I was at it, and this time tried acetone (well, nail-polish remover) to remove the toner. Acetone works much, much better than Scotch-Brite pads.

In the new revision of the board, I made the following changes:
  • VIAs have a larger diameter for the pad-area as my drilling wasn’t terribly accurate and I destroyed many VIAs on the original board.
  • Clearance around many VIAs was improved to reduce the risk of solder bridges.
  • Minor component positioning changes to shorten and simplify routing of traces.
  • Added my name and URL to the top layer (it was already on the bottom layer).

Now all I have to do is get it soldered without any short circuits, and that’s proving to be a big challenge!

V7 Navigation 1000 Menu Button

Shortly after posting the ‘M’ button fix with the steps required if you’d done the USB-port hack in another of my posts, I realized that it was entirely unnecessary to do any hardware modifications to fix the bug – rather you could do it all with just Explorer and a little back-and-forth with the SD card between your GPS and PC.

Anyway, life kept me from posting an update to that, but thankfully someone else has saved me the time! If you head on over to the always-awesome instructables.com, you will find a nice article by dadefatsax which explains all the necessary steps. Unlock V7 Navigation 1000 GPS.

First homemade toner-transfer PCB

A coworker has been raving for a few weeks about how easy and quick it is to make PCBs (printed circuit boards) at home using what is commonly called the “toner transfer” method. Given that the only other way I’ve used to make PCBs at home (using photo-resist coated boards and a UV exposure box) had pretty terrible results and that I wasn’t too eager to drop money on an untested circuit design to get it commercially made for me, it seemed worth the effort to have a go for myself.

Tom Gootee describes the method in incredible detail (if a little disorganized) on his website, Easy PCB Fabrication. I also bought a laser printer for this purpose (well, I wanted a laser printer, and this was a good excuse…) – I decided to go for a Brother HL-5250DN as it has 1200×1200 DPI resolution as I didn’t feel that a 600×600 DPI printer would be able to reproduce the SMD pads and fine traces that I would be using (0.2mm! That’s less than 8mils for those of the non-metric persuasion, or about 4 ‘dots’ at 600 DPI). I didn’t diverge very far from Tom’s instructions as this was my first attempt (ok, second – the first failed to transfer the toner completely to the PCB) with the exception of spending quite a bit more time ironing the design to make sure that the toner transfered this time.

I also picked up an etching kit, heater, and supplies from a local electronics store. Originally I was a bit annoyed when I opened the box to find a regular Tupperware container, an aquarium pump, and a piece of plastic to (poorly) hold the PCB while it etches – however I probably couldn’t have picked up the items individually for too much cheaper anyway, so it wasn’t so bad. I was hoping that the kit would contain a nice tank with graduations marked for the volume of etchant or at least depth though. Instead, the ‘tank’ was quite large – far larger in fact than I’ll ever need – which also means that it requires a lot of etchant to fill it enough to cover even a small PCB. As I did not know this at the time, I picked up only a single 500ml bottle of Ferric Chloride (a common etchant) and as a result had to dilute it 1 part FC to 3 parts water in order to get it to just over the height of my PCB.

At this point I was having serious doubts that I’d see any success as I hadn’t seen anything in any of the PCB making tutorials online about diluting FC. As it turns out, there was no problem other than it taking longer to etch, about 20 minutes I believe (but I wasn’t looking at the clock). I’m sure that the heater and pump helped greatly with this, and without them I probably wouldn’t have seen any results for a much longer period.

So what were the results like for my crazy-tiny traces and SMD pads? Outstanding! Considering 0.2mm is the minimum size for many commercial PCB manufacturers, and even below the minimum for some, I’m very surprised by how well even the tightest areas of the PCB turned out. To the left you can see a closeup of a MLF32 footprint IC with 0.2mm traces and spaces.

The bottom layer of the PCB has a few quite large holes in it, but I think that was mainly due to the fact that I didn’t spend much time ironing that side during the toner-transfer step as it didn’t have many fine traces on it. The holes didn’t break any traces, so that’s fine.

V7 Navigation 1000 Menu-button bug fix

Those who have a V7 Navigation 1000 have probably noticed that the Menu (M) button doesn’t actually go to the menu. It doesn’t do much of anything actually (other than behave as “up” for a few menus), which is quite disappointing since there are only really 3 buttons you can use on the front (excluding the backlight on/off button) and the other two you will usually want to keep at their default functionality (zooming in and out).

Oh hey, what do you know – if you followed the instructions in my last blog entry, you will have access to change this!

  1. Plug the V7 Navigation 1000 into a USB port on your computer and turn it on.
  2. Browse to \Flash Disk\MyGuide\ in Explorer (on the PC, not the GPS).
  3. Copy the 1.3mb Data.zip to your PC’s hard drive.
  4. Rename the Data.zip on the GPS to Data.orig (as a backup).
  5. Open the Data.zip that you copied to your PC and extract config\keybind.txt from the archive.
  6. Open keybind.txt in Notepad and scroll down to the [CARPOINTA1000] section.
  7. Below this, add the following line: UP=”ROUTEINFO”. The section will now look like this:
    [CARPOINTA1000]
    UP=”ROUTEINFO”
    39=”ZOOMIN_DISCRETE”
    37=”ZOOMOUT_DISCRETE”
  8. Save keybind.txt and replace the original keybind.txt in Data.zip with your edited keybind.txt. Make sure that you actually replace the file and that it goes into the correct directory of Data.zip.
  9. Copy the updated Data.zip back to the GPS (put it back in \Flash Disk\MyGuide\).

Done and done! You can now start MyGuide and the Menu (M) button will now behave just like going to Route > Info.

You can actually replace ROUTEINFO with many other useful commands, of which MAINMENU is a good choice (as it is obviously the intended behavior of the Menu button, but I found ROUTEINFO to be more useful personally). Have a look through keybind.txt to get an idea of what other commands you can try.

Of course, if you don’t use the zoom in/out buttons, you can change the bindings on those too.

Jan 5th, 2007 | Filed under V7 and MyGuide