HlslDom is a .NET library for programmatic HLSL shader program creation, inspired loosely by Microsoft's CodeDOM.
When using HlslDom you first create a Program, the top level container for all functions, user defined types, and global variables.
Types that are intrinsic to HLSL, like scalars, vectors, and matrices, are managed by a type registry. These intrinsic types, save samplers, are heirarchical in that vectors are considered a number (1..4) of scalars, and matrices are considered a number (1..4) of vectors. These basic types can then be aggregated by the user into a structure, with semantics set for each field.
HlslDom supports most of the intrinsics that are included up to Shader Model 3.0. Intrinsics can be called like any other function and polymorphic versions will correctly validate parameters and determine return parameters.
The concept at the centre of HlslDom is the expression. Expressions are used to represent everything from arithmetic (with BinaryExprs), to variable declaration (DeclExprs), to program flow (IfExpr/ForExpr/TernaryExpr). Expressions may or may not have a value. Those without values are usually syntactic constructs like if/for/while expressions, while those with value are expressions that evaluate to something that can be used, like a call to another function or a struct member access.
Variables at all scopes can be automatically named, or given names. Expressions that may be expensive to evaluate, like calls to functions or large mathematical constructs, can be assigned to local variables with DeclExprs (declaration expressions), and optionally set to "const", to keep the amount of generated code down.
User-created functions are a collection of at least one expression. That one expression that a function has to have, at a minimum, is a ReturnExpr representing a return value. The returning type of the function is automatically inferred based on the types of its contained ReturnExpr.
Once all the desired user defined functions are created, the program can be emitted. Currently HlslDom supports emitting straight up HLSL or emitting a DirectX effect definition.
HlslDom can be found on GitHub (https://github.com/maxburke/HlslDom).
Monday, January 10, 2011
Fun with XNA, part 2
It's been four months since I've last written about my XNA project -- an eternity! I haven't been idle, most of my work has been behind the scenes on pipeline stuff.
When I left off last I had the world geometry rendered, with basic lightmaps and textures, but there was quite a bit missing. Quake 3 was quite a vibrant game, in large part due to the multitexturing effects that were used. Surfaces animated and swirled, effects were stacked on top of each other, lights flickered and pulsated. If all I'm drawing is polygonal geometry with one texture and maybe a lightmap it's not going to look anywhere near that good.
Quake 3 achieved most of its visual effects with shaders, though being released in 1999 it was many years ahead of languages like HLSL/GLSL/Cg, and even ahead of fragment/vertex assembly languages. Contrast that with today where our world is nearly at a point where el-cheapo feature phones have programmable graphics pipelines. Quake 3's shaders were relatively simple, specifying a texture that could be used, how it was to be sampled and its colours manipulated, and how it was to be blended with others.
I needed a way to get these simple Quake 3 shaders into my engine, into a more XNA-friendly format. There were a few approaches I tried.
First, I tried hand writing a couple shaders. Brute force is occasionally worth it, but not here. It took way too long to be a viable process, but at least I figured out what I would need to do.
Next, I tried generating shaders at runtime but this I shelved because XNA doesn't support runtime shader compilation when deploying applications.
I took the beginnings of the previous step, generating the shaders programmatically, and worked it into a pipeline step. This first step to generating them in a pipeline was to parse the existing Quake 3 shaders. Since the format was documented (here is the Q3 Shader Manual), I figured this was going to be cake. Sadly, it wasn't documented well enough, and I was getting a slew of unexpected tokens and effects. I ended up guessing as to what they might be, within the context they were used, and discarded the rest with a TODO attached that I will take care of them later :).
Once the Q3 shader was parsed I had to turn it into a list of textures, some flags, and a pile of HLSL. The textures and flags were pretty easy, but the HLSL less so. I tried a simple text-based HLSL generator but keeping track of state and variables became messy in short order.
After putting the project down for a few weeks, I was sitting on the couch thinking that it would be nice if it was as easy to build a shader as it was to build C# using Microsoft's CodeDOM. Out of that, I put together a library, HlslDom (which is hosted on GitHub! https://github.com/maxburke/HlslDom), to stitch together HLSL programs via code.
This weekend I was at home, sick, and didn't have much to do but work on HlslDom and this shader compiler. The shader compiler is now done, it generates valid HLSL, and I'm getting the HLSL shaders integrated back into the game portion! I'm hoping to have screenshots following soon.
When I left off last I had the world geometry rendered, with basic lightmaps and textures, but there was quite a bit missing. Quake 3 was quite a vibrant game, in large part due to the multitexturing effects that were used. Surfaces animated and swirled, effects were stacked on top of each other, lights flickered and pulsated. If all I'm drawing is polygonal geometry with one texture and maybe a lightmap it's not going to look anywhere near that good.
Quake 3 achieved most of its visual effects with shaders, though being released in 1999 it was many years ahead of languages like HLSL/GLSL/Cg, and even ahead of fragment/vertex assembly languages. Contrast that with today where our world is nearly at a point where el-cheapo feature phones have programmable graphics pipelines. Quake 3's shaders were relatively simple, specifying a texture that could be used, how it was to be sampled and its colours manipulated, and how it was to be blended with others.
I needed a way to get these simple Quake 3 shaders into my engine, into a more XNA-friendly format. There were a few approaches I tried.
First, I tried hand writing a couple shaders. Brute force is occasionally worth it, but not here. It took way too long to be a viable process, but at least I figured out what I would need to do.
Next, I tried generating shaders at runtime but this I shelved because XNA doesn't support runtime shader compilation when deploying applications.
I took the beginnings of the previous step, generating the shaders programmatically, and worked it into a pipeline step. This first step to generating them in a pipeline was to parse the existing Quake 3 shaders. Since the format was documented (here is the Q3 Shader Manual), I figured this was going to be cake. Sadly, it wasn't documented well enough, and I was getting a slew of unexpected tokens and effects. I ended up guessing as to what they might be, within the context they were used, and discarded the rest with a TODO attached that I will take care of them later :).
Once the Q3 shader was parsed I had to turn it into a list of textures, some flags, and a pile of HLSL. The textures and flags were pretty easy, but the HLSL less so. I tried a simple text-based HLSL generator but keeping track of state and variables became messy in short order.
After putting the project down for a few weeks, I was sitting on the couch thinking that it would be nice if it was as easy to build a shader as it was to build C# using Microsoft's CodeDOM. Out of that, I put together a library, HlslDom (which is hosted on GitHub! https://github.com/maxburke/HlslDom), to stitch together HLSL programs via code.
This weekend I was at home, sick, and didn't have much to do but work on HlslDom and this shader compiler. The shader compiler is now done, it generates valid HLSL, and I'm getting the HLSL shaders integrated back into the game portion! I'm hoping to have screenshots following soon.
Friday, January 7, 2011
Virtuals don't kill performance, people do.
This post is in response to Scott Graham's blog entry on a Data-Oriented C.
There are two areas where we come up short in regards to the use of data oriented programming, tools and (more importantly, IMO) education.
C++ gets a bad rap for enabling poor programming techniques but in the end the language isn't inserting virtuals on our behalf. Evangelism I think is the key here. People need to see the kind of code in action, beyond hand-wavey snippets included in blogs. Books need to be written. School curricula need to be updated to show these techniques to students. There is an entire industry centred around preaching the use of OO patterns and the result is that people finish university thinking that Cats are Animals are Objects and they all need Update methods.
To make the default decision be the correct one it would mean having to make incorrect choices either impossible (preferable) or extremely painful. I think, in that case, that perhaps C isn't the best choice of language as a base for this type of programming. Perhaps a shader-type language, something that is extremely restricted in terms of what the programmer can do outside of his own local task context and ideally removes the task of management of data (memory allocation, etc.) from the actual computation. A language that can be retargeted to asymmetric environments like SPUs or GPUs and operate well within the inherent restrictions that accompany them would be extremely beneficial with the new processors that are entering the market.
There are two areas where we come up short in regards to the use of data oriented programming, tools and (more importantly, IMO) education.
C++ gets a bad rap for enabling poor programming techniques but in the end the language isn't inserting virtuals on our behalf. Evangelism I think is the key here. People need to see the kind of code in action, beyond hand-wavey snippets included in blogs. Books need to be written. School curricula need to be updated to show these techniques to students. There is an entire industry centred around preaching the use of OO patterns and the result is that people finish university thinking that Cats are Animals are Objects and they all need Update methods.
To make the default decision be the correct one it would mean having to make incorrect choices either impossible (preferable) or extremely painful. I think, in that case, that perhaps C isn't the best choice of language as a base for this type of programming. Perhaps a shader-type language, something that is extremely restricted in terms of what the programmer can do outside of his own local task context and ideally removes the task of management of data (memory allocation, etc.) from the actual computation. A language that can be retargeted to asymmetric environments like SPUs or GPUs and operate well within the inherent restrictions that accompany them would be extremely beneficial with the new processors that are entering the market.
Subscribe to:
Posts (Atom)