Old Man Reviews Programming Languages, 2019 Edition

Go

I picked up Go early last year to put together some services for work. There's a lot that I like with the language. It's dead simple; I would bet someone coming from a background in bracey languages can be productive in a couple hours. 

I love being able to get up and running without being able to write any sort of build manifests or scripts or anything, that I can drop in references to third party projects to fill in the few gaps in its massively comprehensive standard library and the out-of-the-box tooling just rolls with it. It has its downsides though, and the ability to manage versions for consistency is a big one. 

Coming from C++ (in games) where exceptions were not an option, and having minor PTSD from reading Herb Sutter's series on So You Think You Can Write Exception Safe Code, and languages like C# where I really had to dig deep to find the effort to handle exceptions, I liked Go's error handling at first. I guess it's that it was explicit about being able to specify "this is a function that returns a value AND an error if something goes wrong"; it seemed more natural than having to see if something threw and how to catch it, or (worse) checked exceptions. I didn't even mind the verbosity.

I like that the go tools build code, run tests on code, format code, and fetch code. My gold standard now in the tabs-vs-spaces war is now "who cares, let the robots handle it".

The compiler error messages are teeerrrrriiibbbble. They are so bad. In some cases it would be an improvement if they just took out the message, leaving the file and line number, and let you dig in. The language itself is simple enough to figure out most errors at a glance.

Rust

I liked Go a lot. Then I tried Rust and now I like Go less.

I've been wanting to dig into Rust for a long time but I've been intimidated by what I've heard. "I'll spend all my time fighting the borrow checker!" Then I had the occasion to have to write a shared object plugin for a service that's live on the internet and suddenly the borrow checker seems like a much better dance partner than trying to harden C or C++.

Yeah, it's a pain, but it's got rules. I spent some time frontloading and internalizing the rules and found that when I started writing code it wasn't a huge battle. 

I spent more time fixing issues with missing semicolons than I did with the borrow checker. I guess a year in Go/TypeScript/Python will do that. 

(But, so far my spelunking into Rust hasn't really taken me deep into managing reference lifetimes.)

Error checking in rust is fantastic. I love that I both have the transparency of the error types you get in Go (ie with Result/Option), but that you can elide the boilerplate with the question-mark operator. Brilliant. Love it.

A lot went into the thought and implementation of the Rust language features and it's a very pleasant, expressive language as a result. Everything is an expression? Love it. Matching? Brilliant. I could go on all day.

Rust's tools put Go's to shame. Like the go tools, they do builds, tests, benchmarks, formatting, package fetching. Something I miss from leaving EA was the code build system. It worked really well for packaged components and I think the closest I've found to this since leaving is Cargo. Package downloads aren't particularly useful in production environments if you can't have consistent environments which is something Go is still figuring out.

(Maybe if you're working at Google and everything is on latest all the time and you have no need to step out of your internal ecosystem it's fine, but one CI system failure on a new and exciting public API change in Dr Janky's Awesome Left Padding Library is too many, because who knows how many behavior changes are included too. But I digress.)

In general the tools just feel polished, like someone has actually internalized that UX is important for more than just web pages. The error messages are top-notch. Providing a way for the compiler to explain in greater detail more of what went wrong? I love it. Microsoft, clang, gcc, take note. Having a link to the warning page on MSDN instead of having to go through Google to get more information would be great. 

The standard library is pretty slim compared to Go, requiring users to have to go out into the wild west to find the functionality they need. It would be nice if more was blessed and included in like a std::ext namespace or something (ie: std::ext::http, ...), but that's not a big deal.

One of the things that I really hated about C++ was it seemed every C++ developer only knew 30% of the language space, but every developer knew a different 30%. And with every iteration of C++ over the last 8 years it seems like both the language space people know is decreasing and the spread of core language concepts is widening. I get the feeling with Rust that there is less conceptual spread in core areas, so while I may only have had exposure to 30% of the language, I'm not finding myself reading things like this C++ value category reference  and wondering why I didn't go into veterinary medicine instead.

If I had a piece of advice to the Rust steering group it would be to not follow C++ down the road of committee driven, lowest common denominator bolt-ons. The language is great. Don't be afraid to extend with both syntax and library (ie: Option/Result/?-operator). Keep doing the great work you're doing!

All that said in praise of Rust over Go, I do want to say that Go is still great for building and deploying tools and services. 

JavaScript / ECMAScript 2015

I mean... it's better? I guess? But... just use TypeScript.

TypeScript

TypeScript is brilliant. Seriously, unambiguously brilliant. I started using it in December so when I noticed that create-react-app supported it. I remember when I first heard that Anders Hejlsberg was starting on TypeScript and was wondering where it would lead; he had done an amazing job with his previous language development and (more importantly) implementation efforts. 

The end result is something that lets you start bringing law and order to the wild west of existing JavaScript code bases, one file at a time. I just started dropping in files written in TypeScript in my existing web front end project and it worked. First time. No screwing around with tools, no nothing.

Brilliant!

Having the sanity of Real Types available when refactoring JavaScript code? Amazing!

Seeing a path forward from the morass of 0 / false / null / undefined? Great!

Being able to use ==/!= instead of ===/!== like a Sane Language(tm)? Where have you been all my life!

Automatic semi-colon insertion without the danger? Hook it to my veins!

A few things surprised me when I started getting into it, like 1) how many existing publicly hosted packages were actually written in TypeScript; 2) how many existing publicly hosted non-TS packages had type annotations available for them; and 3) how I could just drop a module declaration in my index.d.ts file for all modules that didn't fall into categories 1 and 2 and I could get back to work. 

I'd really like to start using in the state backbone of our React front end, ie: the Redux reducers, but where I've tried it I'm finding it's awkward and there's a lot of fighting going on. Maybe I'm doing it wrong. Maybe it's something that's just not meant to be. 

The error messages can be pretty appalling, bringing me back to the days of template metaprogramming with older versions of MSVC. But that's a price I'm willing to pay to reduce the number of times I'm debugging undefined references caused by accessing a misspelled property.

Azure from an AWS user

I've been getting my hands dirty in Azure recently. Until about a month ago all of my cloud experience has been in AWS, but we're sitting on a mountain of unused Azure credits and I think it makes good sense to chip away at these.

1. Service principals are a lot easier to grok and manage than IAM roles

I found it took a long time for me to properly grok IAM roles and use them effectively. Service principals in Azure though seemed to click right away.

2. Blob service not allowing default pages or SSL on custom domains is a huge drag

I really like using S3 for static website hosting. It's simple, easy, and I don't need to manage infrastructure. So, I was hoping that blob storage would fill a similar niche. But it doesn't yet.

Some guides out there made the recommendation to use CDN to make up for the shortfalls in blob storage's functionality, but after playing around with it I wasn't happy. It sort of worked, but the four hour plus turnaround time in configuration changes made iteration hard. I should have realized I was using the wrong tool for the job earlier.

That said, "premium CDN" not offering programmatic expiration/purging of assets is kinda silly

3. Resource groups are awesome!

I like the concept, I like the implementation. It's like the accounting you get with CloudFormation stacks but without having to use CloudFormation.

4. Azure File storage is light years better than EFS

Not like it's a high hurdle to clear but I've been noticing much better performance out of Azure File than from EFS. Easier to get going, no need to manage endpoints, etc.

5. Azure storage has terrible security defaults

Unsecure-by-default may be reasonable if you're using blob storage for static website hosting, but if you're using it for build artifacts then it's terrible.

6. Terraform support is less complete for Azure than AWS

There's things that I find missing in Terraform's support that I keep expecting to be there. For example, a data source representing a load balancer. It doesn't appear to have one, so one of my scripts has an IP hardcoded to the load balancer's IP. 

7. VM image management isn't as slick as AMIs

This may be a tooling thing (ie: Packer/Terraform/Spinnaker) more than an Azure thing; I was really enjoying being able to bake base images in AWS that included all the things I needed. But I can't, for example, seem to bake my own base image and use it in Spinnaker or Packer for building other images.

Using Kubernetes has mostly made this unnecessary though.

8. I miss using accounts for separation of concerns

I like that it's really easy in AWS to create a new account for some set of features. Like, a production account, a staging account, an account for hosting raw data, an account for hosting infrastructure, and yet another one for hosting, well, user accounts.

Azure isn't really architected to make that a possibility.

9. Lack of customizability for Azure Active Directory Domain Services is a drag

I was using the AWS Simple Directory for simple user account management and it was great. One of the things I liked was that I had enough control over directory content to be able to stuff user SSH keys into attributes. This worked really well, it was really easy to manage distribution of keys.

But with AADDS I can't create custom attributes, nor am I able to retrieve content out of what few attributes I can edit. Boo.

In general, some things seem to work better on Azure than AWS, and some things work better on AWS than Azure. Even though the UI of the Azure portal feels more consistent than the AWS console, the underlying features in Azure feel like they're incomplete.

Integrating SSH with AWS Simple Directory

I had a helluva time trying to get SSH to authenticate against AWS Simple Directory (AD), so I figured I'd jot down some notes about what I finally got to work.

First, I added all public SSH keys to the altSecurityIdentities attribute in AD. Probably not what it's for but it's there and it works!

Second, I added a script to /etc/ssh which fetches the public keys for a particular user:

#!/bin/sh
ldapsearch '(&(objectClass=user)(sAMAccountName='"$1"'))' -H ldap://ad.urbanlogiq.com:3268 -D '***' -w '***' 'altSecurityIdentities' | sed -n '/^ /{H;d};/altSecurityIdentities:/x;$g;s/\n *//g;s/altSecurityIdentities: //gp'

... which was referenced by the /etc/ssh/sshd_config file:

[... snip ...]
AuthorizedKeysCommand /etc/ssh/ldap_keys.sh
AuthorizedKeysCommandUser nobody
[... snip ...]

Then, I installed the libpam-ldapd package (note the trailing d!) The nslcd daemon was configured like so:

# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldap://ad.myhouse.inthemiddleofthestreet.com:3268

# The search base that will be used for all queries.
base ***

# The LDAP protocol version to use.
ldap_version 3

# The DN to bind with for normal lookups.
binddn ***
bindpw ***

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com

# SSL options
#ssl off
#tls_reqcert never
tls_cacertfile /etc/ssl/certs/ca-certificates.crt

# The search scope.
scope sub

# Mappings for Active Directory
# This is the important bit; these fields match up with the fields added by Directory Services for UNIX
pagesize 1000
#referrals no
filter passwd (&(objectClass=person))
map    passwd uid              sAMAccountName
map    passwd homeDirectory    "/home/$sAMAccountName"
map    passwd loginShell       "/bin/bash"
map    passwd gecos            displayName
map    passwd uidNumber        objectSid:S-1-5-21-3623811015-3361044348-30300820
map    passwd gidNumber        objectSid:S-1-5-21-3623811015-3361044348-30300820
# If you wish to override the shell given by LDAP, uncomment the next line
#map    passwd loginShell       "/bin/bash"
filter shadow (&(objectClass=person))
map    shadow uid              sAMAccountName
map    shadow shadowLastChange pwdLastSet
filter group  (objectClass=group)
#map    group  gid              member

Note the attribute map above; the uidNumber and gidNumber maps are needed otherwise things will break when it tries to find your user in the default SimpleDirectory schema. Next, nsswitch.conf needs to be updated to use ldap:

[... snip ...]
passwd:         compat ldap
group:          compat ldap
shadow:         compat ldap
[... snip ...]

And, lastly, /etc/pam.d/common-session needs to be augmented for PAM to create the home directory on first login. Just stuff this line at the end:

session  required pam_mkhomedir.so skel=/etc/skel umask=0022

And that's what I did! I still need to sort out sudo functionality. Maybe that'll be a future update.

Software Builds at EA: Sourcing Code

In my previous post I discussed the general overview of EA's build framework, the one with the imaginative name Framework. Relegated to one of the last paragraphs was a mention of the package server and how it is used for enabling sharing of code across teams.

Since that post was written we've added a new mechanism to Framework to share code by means of specifying its location via a URI in the masterconfig file.

The URI sourcer has two back-ends currently implemented, one for fetching code from Perforce, and another for fetching from NuGet.

Perforce URI's encode the server, depot path, and the specific changelist or explicit head revision specifier of the software to pull down. This last condition is meant to improve build reproducability; anyone with a particular master configuration file will have the same code from which to build, local changes notwithstanding.

Because EA is a very distributed company, with studios located across several continents, and because these packages can be large (multi-gigabyte), Framework keeps an internal mapping of central Perforce servers to studio-specific proxy and edge servers, allowing users at these studios to avoid saturating slow trans-Atlantic and trans-Pacific network connections.

As Microsoft moves to distribute more components via NuGet we were looking to add support to Framework to support interoperation. By specifying a NuGet package name, version, and a URI, Framework will fetch the code from the NuGet mirrors. It then installs the package and wraps any contained DLLs in the Framework package scaffolding. This step is not strictly necessary, we could add the package to the MSVC packages.config and be done with it, but it adds a convenience layer to permit any package in the build hierarchy to express dependencies on the NuGet library as if it was a normal Framework package.

This is but the first step in enabling better distributed development workflows. Being able to pull down source code from someone else's server is handy; being able to fork a sub-component package, make a change, share it with your team and/or send a pull request back to the maintainer, while fitting in this same explicit versioning model that Framework relies on and using backing technologies that handle repositories in the tens or hundreds of gigabytes with no problem (trololol), is going to be a lot of fun. Or awesome. I'm hoping for awesome.

Software Builds at EA: The 5000' View

A couple of days ago this tweet by John Carmack popped up. In case the link ever goes away, he says, "Dealing with all the implicit state in the filesys and environment is probably going to be key to eventually making build systems not suck."

At EA we have spent a lot of time developing build systems that don't suck. They're not perfect but as we develop applications and games targeting platforms from Android to Xbox, plus everything in between, they work incredibly well for us.

Framework

The cornerstone of our build infrastructure is a version of NAnt that was forked eons ago and has undergone essentially a complete rewrite. Although the base functionality is still there, and still used, a lot of effort has been put into establishing it as a base for the higher level components.

NAnt is just one piece of a system called Framework. In addition to NAnt, Framework includes features for managing build configurations, SDKs, and software distribution. There are a few core concepts in Framework that I think have made it much easier to manage the build morass: modules, packages, dependencies, masterconfigs, build configuration management, SDK packages, and the package server.

Module

A module is Framework's representation of an artifact producing build process. At a minimum a module will have a type (static/dynamic library, executable, C# assembly, etc.), a name, and a set of input files. Modules have a number of optional components such as custom compile options, custom preprocessor definitions, custom warning suppression, sets of resource files/include files/object files/libraries, and dependencies.

Package

Packages are a collection of modules. Every package in the Framework world has a version, even if that version is "unversioned". A package contains a description of how to build its modules, its public exports (such as public include directories, build artifacts, etc.), as well as a manifest which allows the package author to include structured metadata that can be inspected by the package server, continuous build/test systems, etc.

Dependencies

Dependency handling is one of Framework's killer features. Dependencies can be made onto other modules in the same package, onto other packages, or onto other modules in other packages. For example I can specify that my module has a dependency on EASTL and the build system will ensure that I am using the correct version, build it if it is not already built, and set up the necessary include and library paths.

Masterconfig

Versioned packages lose their utility if there is no way to enforce the versions being used. The masterconfig file is the bill of materials for the application and specifies the version of each dependency, each SDK, and each configuration component.

One of NAnt's script concepts is the property. Essentially a global variable, properties are used for tweaking the state of the build system. Some are set in build files as crude variables, others like the particular configuration to build are set on the command line. A third way to set properties is to encode them in the masterconfig. This ensures that the whole development team is using the same Android SDK level, for example, as the SDK package may provide several.

Most build system state can be controlled in the masterconfig file(*), and checking the masterconfig file into source control can be a easy and effective way of ensuring the team is building with consistent settings.

(*) Provided that the build script writers do not start using environment variables on the side :-)

Build Configuration

Framework also provides global build configurations that describe how to perform a build. These build configurations take high level build goals (ie: optimized 32-bit Windows build) and use it to determine the required compiler invocation command lines.

These build configurations are used as the base for building a project and all of its dependencies. Although a particular dependency may override its build settings for warning suppression, custom optimization settings, etc., for the most part these are slight modifications made to the global configuration.

SDK Packages

SDK packages are not handled any differently by Framework than other packages, but Framework's versioning forces consistent use of SDK versions by all developers. Many SDKs are packaged in ready-to-go forms that the build system will download automatically and use to build, skipping the hassle of having to find the right SDK package and installing it. Or installing it and its numerous components.

Package Server

Where versioned packages come into their own is as a basis for sharing technology between development groups and teams. To make this sharing easier we run a server for people to post their packages on, not entirely unlike Freecode/Freshmeat. If Framework cannot find locally a package listed in the masterconfig it will try to fetch it from the package server.

Less Make, More CMake

Owing to its NAnt heritage Framework has the ability to perform builds itself but with Visual Studio support demanded by most developers it is being used more and more as just a front end for generating project files for Visual Studio or Xcode. 

French Immersion

About this time last year I was traveling from Vancouver to Montreal and eventually to London for work. It wasn't the first time I had done that particular route but it was the first time I drummed up the courage to dust off my awful western Canadian anglophone elective-course high school French and speak la belle langue wherever I went.

Getting into a cab at the airport I asked the driver to take me to my hotel at neuf-cent boulevard Rene Levesque. The driver asked me where I was from, where I was going to, what I was doing in Montreal. In French. I answered, haltingly at first, only slightly less haltingly as old words I had not used for years started to bubble to the surface, punctuating my sentences with apologies for my awful French. He told me his English was worse and encouraged me to practice. We mostly talked about hockey for the rest of the ride.

Searching out dinner I figured I'd keep the momentum I had been building and ordered in French. The waiter responded in English. Dammit.

Two days later I was on the red-eye from Montreal to London. A lady sat down next to me. I asked her in English if she was traveling beyond London. She shook her head apologetically. "No English".

Ou voyages-tu, apres Londres?

"Ah! Syrie!"

It could have been "A Syrie" but I could swear there was an exclamation in the middle.

This was well after the Syrian civil war reduced formerly great cities like Homs to dust and ghosts. She was traveling to see her family whom, she assured me, was neither dust nor ghosts. We talked for the whole plane trip. In French. I think she knew more English than she let on, but she wasn't comfortable speaking it, and I knew no Arabic. We mostly talked about the past, present, and future of Syria.

Time has taken from me many of the the details of our conversation. I do not know where she was traveling within Syria.

After arriving in London I wished her well on the next leg of her trip and thanked her for the opportunity to practice my French.

I hope she was able to get out with her family.

C and C++ need inline directives at the call site

I don't like the inline keyword, I think it's incredibly broken.

Inlining heuristics make reasonable assumptions about what should be inlined but there's basically no control by the user unless they use something like VC++'s __forceinline directive. Automatic inlining works well with trivial functions, the kinds that populate the cctype/ctype.h header, but fails when the desired inline function grows beyond a few statements.

The ability to inline code can be quite a powerful for developers, especially those who had to develop for consoles like the Xbox 360 and PS3, but because the compiler has incomplete information on where the inlining really needs to go and because there's no way in the language to give it better hints, there is often bloat in areas where you don't want it, and no inline expansion in areas where you do.

I've worked around this in the past by abusing macro expansion, but it's not a very clean solution, and makes debugging difficult.

What I would like to see would be an ability to use the inline declaration at the call site so that I can pick and choose where the inline expansion happens as needed. One example I can think of is multiplying two matrices in a video game. In some cases, say in creating the view/projection matrix, an inline expansion of the multiply would be unnecessary code bloat because this operation only typically happens once per game frame. 

Mat4x4 viewProjection = Mat4x4Multiply(view, projection);

While transforming joint hierarchies for animation there are typically a number of multiplications that happen in a tight loop where inlining would be beneficial:

for (int i = 0; i < numBonesInModel; ++i) {

    Mat4x4 transformedJoint = inline Mat4x4Multiply(jointMatrix[i], jointMatrix[parent[i]]);

}

I want more control over what gets inlined and what doesn't, without resorting to macros, and I think being able to specify at the call site what I want inlined would go a long way toward that goal.

Scheme in 5000 lines of C part 5: Collecting garbage

I was hoping to get this post up sooner but the last week and a bit has been complicated by my appendix exploding. I'm on the upswing though and now am starting to think well enough to be able to dig through my Scheme todo list.

Last time this series left off I began working on the garbage collector by adding a test to exhaust the heap and then re-wrote the evaluation engine. So I didn't actually do any work on the garbage collector.

A few years ago I worked on a small team that built a .NET environment for video game consoles which helped spark my interests in compilers and runtime environments. We tried a number of types of garbage collectors while trying to hit our performance goals, from simple non-precise mark/sweep, to Boehm, to several different types of precise collectors. Two designs we had the most success with, one was a half-space (Cheney) collector, and the other final design was based on an algorithm somewhat similar to Clinger's non-predictive generational collector.

At points I have experimented with a generational collector but I found that they work best when you can expand the tenured generation as objects are copied into it. As this isn't really feasible in a video game console where your memory budget is fixed and there's no pagefile to fall back on they never made the cut.

What I found while profiling with the games that were being built on this .NET runtime is that objects either died really quickly or they persisted forever (ie: more than a few frames). By breaking the heap into many small (typically page-sized) buckets, often when a collect was triggered there would be enough buckets that no longer had any live objects that the collect process could finish without having to move/compact the heap at all. This was a huge advantage to the half-space collector that we had used previously because all live objects would be copied in every collection.

What does this have to do with Scheme in 5000 Lines of C? The collector used in this scheme is going to use some similar ideas. For example, it's going to use small, fixed sized buckets. It's also going to compact the heap only if it has to. Luckily this garbage collector doesn't have to worry about the complications that a collector for .NET needs, like finalizers and threading.

I've got the initial root-set scan and collection of empty buckets implemented. The object handles I mentioned last time are now quite important because they act as read-barriers for code outside of the system so that they get an object's correct location if it moves underneath them. It's a bit cumbersome to use in C for now but this is a trade-off I'm making in order to make the collector fast, and enforce to the user (me right now) that it's a bad idea to peek and poke stuff in the scripting system willy-nilly.

One of the most hair tearing-out-ly problems I've found when writing a precise garbage collector is making sure that you do not miss any roots. If you do you'll eventually collect an object's space before it's dead and then you're in world of hurt. The read barrier is an easy way to make sure you do not miss any roots if you do not have the luxury of being able to control code generation so as to generate stack frame maps.

The reader also got a lot of attention today. Sorry, not you :), but the core of the code called when the user calls (read ...).I've weaned it off of the CRT allocation functions and it now allocates all of its intermediate structures in the GC heap. This has been on my todo list for a while.

The source tree was also reworked so as to be a little more friendly for use as a library -- a public include folder was created separate from the library source code.

I've got a few things up next to take care of. I want to start working on closures soon as well as heap compaction. Maybe before the end of the month I can get a few of the Computer Language Benchmark Game benchmarks written and added to the test suite as well.

View the source on GitHub: https://github.com/maxburke/evilscheme

Scheme in 5000 lines of C part 4: (Un)necessary refactorings

Last week I added the first test of the garbage collector which simply allocated memory until the heap was exhausted. Since the garbage collector doesn't do anything now but allocate it was expected that this test would run until the gc_collect function was called where it would hit a breakpoint. The test was simple:


(begin
    (define gc-test (lambda (count param)
        (if (< count 1048576)
            (gc-test (+ 1 count) #(1 2 3 4 5))
            "done")))
    (gc-test 0 0))

It calls itself a million times and allocates a 5-element vector each time. Since the test harness only uses a 1mb heap it should exhaust pretty quickly. (The compiler is pretty stupid currently and doesn't do any sort of optimization that would eliminate this allocation currently).

I ran into one issue first. Up until now, all my tests were testing compilation, so one test case would define a test function and the next would call it. Now, I wanted to do both in a single test case because I'm lazy and don't want to split all my test cases into one where definition happens and another where execution happens.

The system didn't have any ability to handle the (begin ...) syntax. This isn't the end of the world because (begin ...) is pretty easy to implement -- you loop over all the contained forms, evaluating each one, and the result is the value of the last form evaluated -- but it needed to be implemented in two places. Most primitive syntax features like define/set/car/cdr/etc. have an implementation for when they are encountered by the compiler, and an implementation in C for when they are evaluated at the top level. So,

(define foo 3)

and

((lambda () (define foo 3))

would follow two separate implementations. This stemmed from the early roots of "let's see if this will actually work", before there was a compiler.

But I was feeling lazy. I could have implemented begin for both both C code and the VM, but I decided to refactor the evaluation engine so that in the future I only need to implement functionality once. This was done by wrapping every top level function evaluation in an anonymous lambda and ensuring that is run by the VM. The example of (define foo 3) above would be translated into the form below and evaluated.

There were a few other changes that came out of this refactoring. First, procedure object storage is now broken into two parts, a record that stores meta information for the function, the environment in which the function was created, and some function local variable slots. These slots are not local variables in the sense of temporaries during execution but rather storage for objects that the function itself might need to refer to, such as other procedure objects (for closures), raw objects from the parser (ie: from (quote ...)), etc. I think this will make it much easier to tackle.

Second, I had to recognize the eventuality that I'd have to call C functions from the VM and that this new execution model would require it for things like (lambda ...).

Thirdly, I've introduced an object handle. This is meant to be a container that is tracked by the garbage collector so that, if a collection is triggered between two successive calls. Say, for example, you have this code:

struct object_t *foo = gc_alloc(...);

struct object_t *bar = gc_alloc(...);

And the second call to gc_alloc() triggers a collection because the heap was full. If the collector doesn't know about foo and the storage behind that object moves, we're going to be spending weeks finding out what has happened. The handle type is meant to make this more robust by basically placing a read barrier on that reference.

Next up is garbage collection.

(The project is now officially over 6000 lines of content, though only 4200 lines of that is code. That still counts, right?)

Follow on to part 5: Scheme in 5000 lines of C part 5: Collecting garbage
View the source on GitHub: https://github.com/maxburke/evilscheme

Scheme in 5000 lines of C part 3: Testing, random thoughts on closures.

I haven't made as much progress on the Scheme recently as I would have liked. Work's been busy, the house has needed some fixing, and I'm still chewing over how I want to implement closures.

Since I want to make *some* progress I rewrote the testing framework. Before I just had some sample strings embedded in main that I read-evaled-printed, and looked at the output to see if the changes I made had the desired effect.

I felt some (a lot of) shame because (semi-)robust testing is supposed to be my day job. So I moved most of that away into a test harness that's pretty stupid/simple but actually performs validation. Funny that.

There's a directory, tests/, that contains the test cases. One per file. Each file has the input that is fed to read-eval-print, the output of which is captured, and then compared to the expected value.

I'm still not quite sure how I'm going to handle closures though. Variable capture I think is going to be fairly easy. I'm not quite sure how I'm going to store references to inner functions though. For example, this function:


(define closure-test
  (lambda (foo)
    (lambda ()
      (set! foo (+ 1 foo))
      foo)))

The outer function (closure-test) will need to hold some sort of managed reference to the inner function so that when closure-test is called it can create a new function object and a new environment, but I'm not quite sure if I want to bake the reference into the bytecode, or do something else. Maybe re-write function objects to include a table of contained functions? That might work.

Follow on to part 4: Scheme in 5000 lines of C part 4: (Un)necessary refactorings
View the source on GitHub: https://github.com/maxburke/evilscheme

Scheme in 5000 lines of C part 2: (let ((them (eat 'cake))))

I've spent the last week implementing the Scheme binding constructs let and let* for my little toy Scheme (https://github.com/maxburke/evilscheme). In Scheme, as with Common Lisp, the difference between let and let* is that let* ensures the bindings are evaluated left-to-right whereas for let they are unspecified. So, if you have let*, you should have let. Easy peasy! (If I'm wrong please let me know!)

The binding constructs surprisingly easy to implement. The VM which started out as purely stack-based (ie, (+ 1 1) became PUSH 1, PUSH 1, ADD) now has instructions for loading from, and storing to, basically arbitrary locations on the stack. The compiler now creates a spot for the active local variables and loads from/stores to those stack slots as required. The slots then are recycled as scopes are exited/entered. On my list for future refactoring is merging the opcodes used to load from/store to stack slots with the opcodes used to load from/store to argument slots because it's the same thing now.

The compiler -- so far -- is sort-of-multi-pass. It does a single pass over the source AST and does a rough compilation. After it performs some clean-up passes on the generated bytecode.

One pass removes the nops that are inserted into the bytecode stream as a result of branching. One difficulty I ran into when compiling conditional branches is that in many cases the branch targets weren't yet known, and when they were known it was a lot of messy code to find where the actual branch target was. The workaround I picked was to insert nops as branch targets, and then use these nops as the first instruction of the resulting basic blocks. It sounds kinda convoluted but it was much cleaner and straight forward to implement. This nop-removal pass, well, removes the nops and updates the branch instruction targets accordingly.

Another pass inspects all branch targets and if the instruction is a return, it replaces the branch with a return instruction. This is the only optimization pass. So far.

The last pass promotes all tail calls to use the VM's tailcall opcode. Scheme has a requirement that implementations must be tail recursive and, although I'm not gunning for 100% standards adherence, I wondered how difficult it would be to add. This pass takes calls that could be tail calls (basically any instruction sequence of a call proceeded immediately by a return) and replaces it with a tail call. The VM then ensures that these tail calls are executed in constant space.

I'm going to tackle closures next because it's been something on my want-to-do list for a while. After that I'll probably tackle the garbage collector, alternative lambda syntax (ie, (define (foo a b c)) in addition to (define foo (lambda (a b c)))), interactive REPL + error callbacks instead of just assertions, benchmarking, and proper quasi-quotation. I'm not sure what I'll do with continuations, that part of the spec I may skip entirely because I'm too dumb to understand how call/cc actually works, but some sort of co-routine mechanism will probably end up going in.

Follow on to part 3: Scheme in 5000 lines of C part 3: Testing, random thoughts on closures.
View the source on GitHub: https://github.com/maxburke/evilscheme

Scheme in 5000 lines of C

Every time I see a blog post spring up talking about a new (basic) implementation of Scheme that is written in 30 lines of Ruby, Javascript, or Python I get the sense that there's something missing.

In fact, there's a lot that's missing.

These environments already come with a garbage collector, so the implementation never describes implementation. The intricacies and tradeoffs of object storage? Also skipped over. Parsing? Nope. What about compilation? Never done.

I appreciate code golf and it's fun to see these new, super small, implementations show up, and don't mean to belittle the effort their developers invested, however I haven't been able to learn much from them. So, I thought I'd give it a go on my own.

The hard way.

In C.

Not that cutesy C with C++ comments, variable-declaration-wherever-you-want-it, or inline functions, but the super crusty C89 supported by Microsoft Visual C++, the one where Dave Cutler personally wrote the front end himself while fighting off the red menace and arm wrestling bears. (And GCC/clang too... I guess. Twist my rubber arm.)

I've got a bunch of stuff cobbled together now where it sorta kinda works. It parses, it generates an AST, it compiles the AST into a bytecode, and the bytecode runs. Hey, that almost makes it more functional than some video game console toolchains I've used.

One decision I made recently which simplified the VM conceptually was creating value types (fixnum/flonum/bool/char/reference/inner reference) from reference types (cons cell/vector/string/function). This allowed the VM's execution stack to contain either a raw value, like a fixnum, or a reference to a heap-allocated object, making stack operations simple.

Originally I had it so that the storage of conses was handled by holding a pair of pointers to other objects but I recently switched this so that a cons is now a vector of length two. This means that value types do not need to be boxed in order to be stored in a cons and also simplifies the operations on it.

I've got a few compiler-related tasks I'm going to work on now that it's executing, such as handling specific essential syntax forms. I've got a garbage collector to do as well, but I'm not worried about that one as I've already written several, including one that shipped in a console video game.

If you want to follow along, I've got my code stashed on GitHub. It's still pretty rough around the edges though, so beware at this point!

Maybe when (or if!) I finish, it'll actually be under 5000 lines of C :-)

Follow on to part 2: Scheme in 5000 lines of C part 2: (let ((them (eat 'cake))))

New job!

I started a new job last month! I'm managing my department's just-formed test engineering group. The group has eight people in both Burnaby and Montreal and we're responsible for developing and maintaining our build farm, performing integration testing, developing/implementing new (for the department) testing processes, and auditing current test coverage. Basically if it's related to testing, and it's something more complicated than running the unit tests, this group will be involved.

I'm really excited about the opportunities that are on the horizon. I started at EA over 10 years ago as a game tester and started as a developer in this group's long dead predecessor doing clear box software testing, so it's in a way like coming home. Since it's removed from the secret squirrel work that I was working on before I will also be a lot more free to talk about what I'm doing.

The first couple weeks on the job were spent getting to know the people, getting to know the work, and setting up a plan where I want the group to go over the next 12 months. As the people on the team aren't new, just repurposed from other groups, most of the work they do in the short term isn't going to change. Longer term the goal is to have detach from their old teams so that they can focus more on the work our department does as a whole.

I'm not going to give up coding quite yet. I've currently got my sights set on our build farm and its reporting infrastructure. It's been cobbled together out of a number of shell/build scripts and it's creaking. One of the more recent additions to the group, Dave, is tackling both this and the database back end to make the system more reliable and easier to report on.

I hate SharePoint.

The web dashboard is built on SharePoint. I suppose there's some sort of redeeming quality I haven't yet seen because apparently a lot of people pay a lot of money for the product but deep down I sincerely doubt it. Anyways, the rationale of our build web dashboard using SharePoint is that there is a need to keep some platforms hidden from the world while they are covered by NDAs -- but the rest of the baggage SharePoint has brought means it takes 20 seconds to retrieve the "current status" web page. If you want to filter this information -- which is pretty common for most people as our group supports 150+ libraries built in 40+ configurations, making a status table pretty useless to get your desired information -- it hits the server again, which means waiting another 20 seconds.

I despise SharePoint.

I've started the process of removing Sharepoint completely and instead use some simple scripts to pull the information from the database and authenticate it via LDAP. I moved the rendering client-side, using jQuery and Bootstrap for a little style. Page load is now down to ~2 seconds with most of it being spent in database queries, and that should remedy itself when Dave's finished his schema rework.

The site is much easier to work on now. Deploying the new pages is handled by syncing them from Perforce. Since all the UI code is in Javascript most of it can be tweaked locally on the desktop without having to setup a local server. Or find the one machine that we have a SharePoint developer environment installed on and without colliding with someone else who also needs to use it, and then package+deploy your changes to the production servers hoping that the servers don't barf on your changes leading you to get a nastygram from IT.

Have I mentioned that I hate SharePoint?

I hope to finish the web page updates this week. Once that's done I have to work on a metrics solution for our console unit tests and dig back into valgrind.

The One-on-One

Any developer worth hiring will tell you that the best meeting is the meeting that doesn't happen. Some will even be so kind as to tell you that at the beginning of every meeting they're invited to!

The one meeting that I always get value from, however, is the one-on-one. Whether or not I'm having the meeting with my manager or someone on my team I'll always come away with something that made the time invested worth while..

The one-on-one is a meeting where there may be venting, there may be difficult conversations, there may be awkward silence, but the manager's job isn't to be defensive or aggressive, it's to play the part of a good listener. If there is a slip up then the trust will be damaged, if not broken entirely, and will take a long time to repair. This isn't the trust that's related to integrity, such as failing to meet a promised deadline, but the "I won't tell you that I'm unhappy with what's going on and I'm looking for new jobs on Monster.com when you're not around" kind.

Although work status will frequently come up during the one-on-one I make it a point to never explicitly put it on the agenda. I want the meeting to be about the status of Fred, not the status of  Fred's work, which means the first question I always lead with is "How are you doing?". I'll also ask some variant of "what's getting in your way and what's on your mind?", which has a subtle, but key, distinction from being work status related as this could lead beyond a discussion of the issue that he's blocked on and into one about how he'd really like to get two days off to help his girlfriend move but he's not sure about taking it because of the amount of work that's piling up. Their response to that question leads quite naturally into "what can I do to help you?"

I've never encountered a programmer that has said "I really like that this meeting was booked in the middle of the afternoon, I really didn't need to be concentrating on finding that one rare concurrency bug." I could be wrong.


The team is being paid to do good work, and I'm being paid to shelter them from BS, so in order to shelter them from BS and allow them to do good work I try to keep meetings to a minimum and preferably at the periphery of the day. Putting a meeting in the middle of the day kills what your team members are working on, causing them to take a while to get back up to speed once it has concluded. I try to put my one-on-ones at the beginning or end of a solid work block, such as the beginning of their (not your!) day, and either before or after lunch, typically avoiding the end of the day because the end of the day can vary with whatever is going on. A good read on this topic is Paul Graham's essay, Maker's Schedule, Manager's Schedule.

Co-routines in C(++)

A little while ago I put together a small co-routine library in C targeting x86/Windows. It uses micro-stacks of about 4kb, minus a small tax for library overhead, for each co-routine which places some constraints on the work that can be done but I figured that it'd complement Windows' fairly heavyweight fiber library.

The code sets up a new thread context, by setting esp to point to the new co-routine stack, and then uses setjmp and longjmp to yield the co-routine to the main execution context and to resume the co-routine from where it left off.

One known issue is that if you build C++ code with Visual C++'s /EH command line options the compiler will automatically run destructors on stack objects as it unwinds past them, and so any objects you create in your co-routines will be destroyed prematurely. Building without exceptions enabled will work around this.

Code: https://github.com/maxburke/coroutine

Management 102: Post shake-up stability

My last post mentioned a few points to keep in mind now that you're a fresh manager. In this one I'm going to go into the first tasks you need to accomplish in first day after the reorganization has been announced. The reorganization doesn't have to be a full company shake up, it could be something just as simple as two people shuffling around.

Hopefully you've had some advance notice that this was coming, giving you some time to plan things out, knowing who will be reporting to you, and possibly what you'll be working on. (Digression: I dislike the term "report" in the management-hierarchy-sense. I guess it comes out of industries where there's a more rigid command-and-control type of infrastructure. Like the military. But the people in your group aren't soldiers; no, you're paying them good money so that they can flex that brain of theirs, not just follow orders. The company's paying you to be their BS shield.)

Re-orgs are kept on the down-low to keep the hand wringing to a minimum, but see if you can suss out who else is in the know, and if they are currently managing some of your new group. Same goes with the new projects you'll be overseeing. The goal here is that once day one arrives you'll look like you have some clue as to what's going on and who these people are. You'll want to sort out a transition plan for your current responsibilities so that you don't appear to have bolted on your old job.

If you've got some time to kill before the big day you might want to start planning your schedule, where you'll fit in your new one-on-one meetings. Getting your one-on-one's setup is a great first step as this is where you put your finger on the pulse of the team; far beyond a face-to-face status report it's where you find out what's on their mind, what troubles may be lurking below the surface. I believe that the one-on-one is so important that I'll go into it in detail in my next entry.

It's a good idea to also have some plan of what you want them to do in the short term, until you can really start digging into the new problem space, even if the plan is to keep on keepin' on. Don't let them twist in the wind.

Once the one-on-ones have been sorted out and you've met them, it's probably a good time to get the team together for a lunch, to get away from the work and get to know each other a bit.

As the day winds down you should start seeing some semblance of routine; maybe you'll have your first official meetings with the team. Your old job should be fading from view, as you'll have plenty to do with your new team and you don't want to give them the impression that they're not the most important part of your work life.

In the end you want to make sure that there's no large clouds of uncertainty hanging over your team's heads. Don't sweat it, you're going to do great!

Management 101

You're a lead programmer and have been for a number of months, or years, and one day your manager taps you on the shoulder and says that there's going to be a change.

You're going to be a manager!

Congratulations!

Welcome to management!

Your work life will never be the same! Well, that's a bit of an exaggeration but things will be quite different. You've had a taste of leadership and mentorship leading up to now (right? ...right?!), so the difference won't be so stark.

I'm not going to talk about management styles so much as point out a few things that may not be obvious for programmers who are new to managing people instead of software.

Obvious-yet-not-obvious point #1: Managers are information conduits.

Your primary goal is to get all the information to your team that it needs in a timely manner. You need to beat the rumor mill; every time they look elsewhere for information, their faith in you degrades a tiny bit. Think about when you go to your manager most often -- it's not to ask about your raise (well, maybe it is...), but it's to know what's going on, who needs what by when, what's coming down the pipe next month, or who should be taking care of that urgent support issue that just fell out of the sky.

Obvious-yet-not-obvious point #2: Nothing should be a(n unpleasant) surprise when it doesn't have to be.

Pleasant surprises are great. "Beer this Friday? Company's paying? Sweet!"

Unpleasant surprises aren't. Some are occasionally unavoidable -- the company often can't give advance warning about layoffs because that will wreak unnecessary havoc on morale and the share price -- but many aren't. If you're about to give someone a below target performance evaluation it should not be a shock to them. The lead up to this should have included several, possibly difficult, conversations about where things are heading and the consequences if the trend continues. Not only that but a case like that should have a plan for remediation.

Obvious-yet-not-obvious point #3: You are responsible for the growth of your people.

One of the big differences of being in a management role compared to being in a mentorship-focused lead role is that when one of your employees didn't get a promotion for which they had been hoping, you are now the first person they will ask why. And you better have good answers for them.

Every quarter I sit down with each member of the team and we put together a list of accomplishments that we want to achieve over the following three months. This list of objectives is broken down into what's expected (on target) and what goes above and beyond expected (above target/significantly above target). At the end of the quarter we go over everything that's been accomplished, and at the end of the review period it's obvious to all parties if there's been continuous, steady improvement.

Although the onus to execute lies with them, you need to coach them, guide them, find opportunities for them and help them find their way out of ruts.

Obvious-yet-not-obvious point #4: Trust is difficult to form but easy to break.

One item to keep in mind is that topics that come up in a 1-on-1 meeting should be considered confidential. If you ask your star why he seems distracted and not his usual affable self and he mentions that he's dealing with some unpleasant family issues, you'll want to keep those discussion notes private. "Jim's not on his A-game because his Mom died" is not an appropriate point to mention at the team stand-up meeting. It's also not appropriate to mention if anyone else asks why Jim seems to not be his usual self.

During that conversation, a lot of faith and trust was placed in you when that news was broken, it may have taken them a long time to work up to telling someone at work about it, and if you violate that trust you may never see it again. This may mean that you'll never be the recipient of other personal thoughts, like "I'm unhappy with my job and want to do something else, probably somewhere else."

Obvious-yet-not-obvious point #5: You'll have less time to do programming at work, but that's OK.

Much of your schedule is going to disappear to meetings, to the overhead of task scheduling, to problem solving, and back to meetings. At first I felt that I was failing because I was doing less programming work but this wasn't the case -- it just took a while to recognize that all the other tasks filling my schedule had value too.

Obvious-yet-not-obvious point #6: People aren't machines, they don't have instruction manuals.

You can't attach a debugger to one of your employees to find out what's wrong, nor can you write an automated test to prevent problems from coming up. Forming a network with your current and former managers is extremely helpful when it comes to fielding these new problems that you will run into. Failing that, you can always ask HR, they are going to become surprisingly helpful in your new role.

I hope that I didn't scare you -- you're going to do fine. Really. Some adjustment will be needed but it'll all smooth out soon enough. In my next post I'm going to talk about establishing stability after this minor organization shake-up.

A Non-Exhaustive List Of Things That Xcode Sucks At

I do a lot of cross platform development, ranging from PC to consoles to Mac/iOS, and as a result I've used a lot of terrible tools before. Some were early in the development cycle of a new platform so they were understandably rough around the edges. Xcode, however, is a whole new level of terrible. Device registration and provisioning is a pain but this is mitigated by not having to do it all that often.

• It crashes. A lot.
• No mixed source/assembly debugging. Having source interspersed with assembly makes it much easier to find your bearings and orient yourself with respect to your code. Mind you mixed listings shouldn't replace either source-only or assembly-only debugging, but should complement them.
• Registers view is at the bottom of the watch window. If I'm in assembly land I don't want to see my symbolic locals, I want to see my registers. Allow me to put my registers front and center!
• Focus moves from debugger command window when "step instruction" commands are issued from outside my own code. I have to re-click on the debugger command window every time I punch in "si". On what planet is this acceptable?
• The debug menu says that control-F7 will step one instruction. Pressing control-F7 on my keyboard (ultra thin USB) does not step one instruction. In fact, it doesn't do anything.
• Hovering over the debug icons say that pressing control and clicking "step into" will step a single instruction. But this isn't a sticky setting, so if I press control and click "step into" twice, the first time will step an instruction and the second will perform a source line step. Mondo fail.
• It would be nice if the callstack window was separate from the thread selection control. This way it's quick to cycle among multiple threads to get an idea of where each one is without having to expand, and contract, multiple drop-downs.
• Tabs with source files would be *super* *handy* for switching back-and-forth between multiple files when debugging. Often I find my self changing (enabling/disabling) breakpoints in separate files regularly as execution progresses and it's a huge pain to have to go back to the folder view to find my file.

The principle of least astonishment is a good guideline for putting together a good user experience, unfortunately whenever I have to use Xcode in anger it consistently astonishes me.

HlslDom: An Overview

HlslDom is a .NET library for programmatic HLSL shader program creation, inspired loosely by Microsoft's CodeDOM.

When using HlslDom you first create a Program, the top level container for all functions, user defined types, and global variables.

Types that are intrinsic to HLSL, like scalars, vectors, and matrices, are managed by a type registry. These intrinsic types, save samplers, are heirarchical in that vectors are considered a number (1..4) of scalars, and matrices are considered a number (1..4) of vectors. These basic types can then be aggregated by the user into a structure, with semantics set for each field.

HlslDom supports most of the intrinsics that are included up to Shader Model 3.0. Intrinsics can be called like any other function and polymorphic versions will correctly validate parameters and determine return parameters.

The concept at the centre of HlslDom is the expression. Expressions are used to represent everything from arithmetic (with BinaryExprs), to variable declaration (DeclExprs), to program flow (IfExpr/ForExpr/TernaryExpr). Expressions may or may not have a value. Those without values are usually syntactic constructs like if/for/while expressions, while those with value are expressions that evaluate to something that can be used, like a call to another function or a struct member access.

Variables at all scopes can be automatically named, or given names. Expressions that may be expensive to evaluate, like calls to functions or large mathematical constructs, can be assigned to local variables with DeclExprs (declaration expressions), and optionally set to "const", to keep the amount of generated code down.

User-created functions are a collection of at least one expression. That one expression that a function has to have, at a minimum, is a ReturnExpr representing a return value. The returning type of the function is automatically inferred based on the types of its contained ReturnExpr.

Once all the desired user defined functions are created, the program can be emitted. Currently HlslDom supports emitting straight up HLSL or emitting a DirectX effect definition.

HlslDom can be found on GitHub (https://github.com/maxburke/HlslDom).

Fun with XNA, part 2

It's been four months since I've last written about my XNA project -- an eternity! I haven't been idle, most of my work has been behind the scenes on pipeline stuff.

When I left off last I had the world geometry rendered, with basic lightmaps and textures, but there was quite a bit missing. Quake 3 was quite a vibrant game, in large part due to the multitexturing effects that were used. Surfaces animated and swirled, effects were stacked on top of each other, lights flickered and pulsated. If all I'm drawing is polygonal geometry with one texture and maybe a lightmap it's not going to look anywhere near that good.

Quake 3 achieved most of its visual effects with shaders, though being released in 1999 it was many years ahead of languages like HLSL/GLSL/Cg, and even ahead of fragment/vertex assembly languages. Contrast that with today where our world is nearly at a point where el-cheapo feature phones have programmable graphics pipelines. Quake 3's shaders were relatively simple, specifying a texture that could be used, how it was to be sampled and its colours manipulated, and how it was to be blended with others.

I needed a way to get these simple Quake 3 shaders into my engine, into a more XNA-friendly format. There were a few approaches I tried.

First, I tried hand writing a couple shaders. Brute force is occasionally worth it, but not here. It took way too long to be a viable process, but at least I figured out what I would need to do.

Next, I tried generating shaders at runtime but this I shelved because XNA doesn't support runtime shader compilation when deploying applications.

I took the beginnings of the previous step, generating the shaders programmatically, and worked it into a pipeline step. This first step to generating them in a pipeline was to parse the existing Quake 3 shaders. Since the format was documented (here is the Q3 Shader Manual), I figured this was going to be cake. Sadly, it wasn't documented well enough, and I was getting a slew of unexpected tokens and effects. I ended up guessing as to what they might be, within the context they were used, and discarded the rest with a TODO attached that I will take care of them later :).

Once the Q3 shader was parsed I had to turn it into a list of textures, some flags, and a pile of HLSL. The textures and flags were pretty easy, but the HLSL less so. I tried a simple text-based HLSL generator but keeping track of state and variables became messy in short order.

After putting the project down for a few weeks, I was sitting on the couch thinking that it would be nice if it was as easy to build a shader as it was to build C# using Microsoft's CodeDOM. Out of that, I put together a library, HlslDom (which is hosted on GitHub! https://github.com/maxburke/HlslDom), to stitch together HLSL programs via code.

This weekend I was at home, sick, and didn't have much to do but work on HlslDom and this shader compiler. The shader compiler is now done, it generates valid HLSL, and I'm getting the HLSL shaders integrated back into the game portion! I'm hoping to have screenshots following soon.