Old Man Reviews Programming Languages, 2019 Edition

Go

I picked up Go early last year to put together some services for work. There's a lot that I like with the language. It's dead simple; I would bet someone coming from a background in bracey languages can be productive in a couple hours. 

I love being able to get up and running without being able to write any sort of build manifests or scripts or anything, that I can drop in references to third party projects to fill in the few gaps in its massively comprehensive standard library and the out-of-the-box tooling just rolls with it. It has its downsides though, and the ability to manage versions for consistency is a big one. 

Coming from C++ (in games) where exceptions were not an option, and having minor PTSD from reading Herb Sutter's series on So You Think You Can Write Exception Safe Code, and languages like C# where I really had to dig deep to find the effort to handle exceptions, I liked Go's error handling at first. I guess it's that it was explicit about being able to specify "this is a function that returns a value AND an error if something goes wrong"; it seemed more natural than having to see if something threw and how to catch it, or (worse) checked exceptions. I didn't even mind the verbosity.

I like that the go tools build code, run tests on code, format code, and fetch code. My gold standard now in the tabs-vs-spaces war is now "who cares, let the robots handle it".

The compiler error messages are teeerrrrriiibbbble. They are so bad. In some cases it would be an improvement if they just took out the message, leaving the file and line number, and let you dig in. The language itself is simple enough to figure out most errors at a glance.

Rust

I liked Go a lot. Then I tried Rust and now I like Go less.

I've been wanting to dig into Rust for a long time but I've been intimidated by what I've heard. "I'll spend all my time fighting the borrow checker!" Then I had the occasion to have to write a shared object plugin for a service that's live on the internet and suddenly the borrow checker seems like a much better dance partner than trying to harden C or C++.

Yeah, it's a pain, but it's got rules. I spent some time frontloading and internalizing the rules and found that when I started writing code it wasn't a huge battle. 

I spent more time fixing issues with missing semicolons than I did with the borrow checker. I guess a year in Go/TypeScript/Python will do that. 

(But, so far my spelunking into Rust hasn't really taken me deep into managing reference lifetimes.)

Error checking in rust is fantastic. I love that I both have the transparency of the error types you get in Go (ie with Result/Option), but that you can elide the boilerplate with the question-mark operator. Brilliant. Love it.

A lot went into the thought and implementation of the Rust language features and it's a very pleasant, expressive language as a result. Everything is an expression? Love it. Matching? Brilliant. I could go on all day.

Rust's tools put Go's to shame. Like the go tools, they do builds, tests, benchmarks, formatting, package fetching. Something I miss from leaving EA was the code build system. It worked really well for packaged components and I think the closest I've found to this since leaving is Cargo. Package downloads aren't particularly useful in production environments if you can't have consistent environments which is something Go is still figuring out.

(Maybe if you're working at Google and everything is on latest all the time and you have no need to step out of your internal ecosystem it's fine, but one CI system failure on a new and exciting public API change in Dr Janky's Awesome Left Padding Library is too many, because who knows how many behavior changes are included too. But I digress.)

In general the tools just feel polished, like someone has actually internalized that UX is important for more than just web pages. The error messages are top-notch. Providing a way for the compiler to explain in greater detail more of what went wrong? I love it. Microsoft, clang, gcc, take note. Having a link to the warning page on MSDN instead of having to go through Google to get more information would be great. 

The standard library is pretty slim compared to Go, requiring users to have to go out into the wild west to find the functionality they need. It would be nice if more was blessed and included in like a std::ext namespace or something (ie: std::ext::http, ...), but that's not a big deal.

One of the things that I really hated about C++ was it seemed every C++ developer only knew 30% of the language space, but every developer knew a different 30%. And with every iteration of C++ over the last 8 years it seems like both the language space people know is decreasing and the spread of core language concepts is widening. I get the feeling with Rust that there is less conceptual spread in core areas, so while I may only have had exposure to 30% of the language, I'm not finding myself reading things like this C++ value category reference  and wondering why I didn't go into veterinary medicine instead.

If I had a piece of advice to the Rust steering group it would be to not follow C++ down the road of committee driven, lowest common denominator bolt-ons. The language is great. Don't be afraid to extend with both syntax and library (ie: Option/Result/?-operator). Keep doing the great work you're doing!

All that said in praise of Rust over Go, I do want to say that Go is still great for building and deploying tools and services. 

JavaScript / ECMAScript 2015

I mean... it's better? I guess? But... just use TypeScript.

TypeScript

TypeScript is brilliant. Seriously, unambiguously brilliant. I started using it in December so when I noticed that create-react-app supported it. I remember when I first heard that Anders Hejlsberg was starting on TypeScript and was wondering where it would lead; he had done an amazing job with his previous language development and (more importantly) implementation efforts. 

The end result is something that lets you start bringing law and order to the wild west of existing JavaScript code bases, one file at a time. I just started dropping in files written in TypeScript in my existing web front end project and it worked. First time. No screwing around with tools, no nothing.

Brilliant!

Having the sanity of Real Types available when refactoring JavaScript code? Amazing!

Seeing a path forward from the morass of 0 / false / null / undefined? Great!

Being able to use ==/!= instead of ===/!== like a Sane Language(tm)? Where have you been all my life!

Automatic semi-colon insertion without the danger? Hook it to my veins!

A few things surprised me when I started getting into it, like 1) how many existing publicly hosted packages were actually written in TypeScript; 2) how many existing publicly hosted non-TS packages had type annotations available for them; and 3) how I could just drop a module declaration in my index.d.ts file for all modules that didn't fall into categories 1 and 2 and I could get back to work. 

I'd really like to start using in the state backbone of our React front end, ie: the Redux reducers, but where I've tried it I'm finding it's awkward and there's a lot of fighting going on. Maybe I'm doing it wrong. Maybe it's something that's just not meant to be. 

The error messages can be pretty appalling, bringing me back to the days of template metaprogramming with older versions of MSVC. But that's a price I'm willing to pay to reduce the number of times I'm debugging undefined references caused by accessing a misspelled property.

Azure from an AWS user

I've been getting my hands dirty in Azure recently. Until about a month ago all of my cloud experience has been in AWS, but we're sitting on a mountain of unused Azure credits and I think it makes good sense to chip away at these.

1. Service principals are a lot easier to grok and manage than IAM roles

I found it took a long time for me to properly grok IAM roles and use them effectively. Service principals in Azure though seemed to click right away.

2. Blob service not allowing default pages or SSL on custom domains is a huge drag

I really like using S3 for static website hosting. It's simple, easy, and I don't need to manage infrastructure. So, I was hoping that blob storage would fill a similar niche. But it doesn't yet.

Some guides out there made the recommendation to use CDN to make up for the shortfalls in blob storage's functionality, but after playing around with it I wasn't happy. It sort of worked, but the four hour plus turnaround time in configuration changes made iteration hard. I should have realized I was using the wrong tool for the job earlier.

That said, "premium CDN" not offering programmatic expiration/purging of assets is kinda silly

3. Resource groups are awesome!

I like the concept, I like the implementation. It's like the accounting you get with CloudFormation stacks but without having to use CloudFormation.

4. Azure File storage is light years better than EFS

Not like it's a high hurdle to clear but I've been noticing much better performance out of Azure File than from EFS. Easier to get going, no need to manage endpoints, etc.

5. Azure storage has terrible security defaults

Unsecure-by-default may be reasonable if you're using blob storage for static website hosting, but if you're using it for build artifacts then it's terrible.

6. Terraform support is less complete for Azure than AWS

There's things that I find missing in Terraform's support that I keep expecting to be there. For example, a data source representing a load balancer. It doesn't appear to have one, so one of my scripts has an IP hardcoded to the load balancer's IP. 

7. VM image management isn't as slick as AMIs

This may be a tooling thing (ie: Packer/Terraform/Spinnaker) more than an Azure thing; I was really enjoying being able to bake base images in AWS that included all the things I needed. But I can't, for example, seem to bake my own base image and use it in Spinnaker or Packer for building other images.

Using Kubernetes has mostly made this unnecessary though.

8. I miss using accounts for separation of concerns

I like that it's really easy in AWS to create a new account for some set of features. Like, a production account, a staging account, an account for hosting raw data, an account for hosting infrastructure, and yet another one for hosting, well, user accounts.

Azure isn't really architected to make that a possibility.

9. Lack of customizability for Azure Active Directory Domain Services is a drag

I was using the AWS Simple Directory for simple user account management and it was great. One of the things I liked was that I had enough control over directory content to be able to stuff user SSH keys into attributes. This worked really well, it was really easy to manage distribution of keys.

But with AADDS I can't create custom attributes, nor am I able to retrieve content out of what few attributes I can edit. Boo.

In general, some things seem to work better on Azure than AWS, and some things work better on AWS than Azure. Even though the UI of the Azure portal feels more consistent than the AWS console, the underlying features in Azure feel like they're incomplete.

Integrating SSH with AWS Simple Directory

I had a helluva time trying to get SSH to authenticate against AWS Simple Directory (AD), so I figured I'd jot down some notes about what I finally got to work.

First, I added all public SSH keys to the altSecurityIdentities attribute in AD. Probably not what it's for but it's there and it works!

Second, I added a script to /etc/ssh which fetches the public keys for a particular user:

#!/bin/sh
ldapsearch '(&(objectClass=user)(sAMAccountName='"$1"'))' -H ldap://ad.urbanlogiq.com:3268 -D '***' -w '***' 'altSecurityIdentities' | sed -n '/^ /{H;d};/altSecurityIdentities:/x;$g;s/\n *//g;s/altSecurityIdentities: //gp'

... which was referenced by the /etc/ssh/sshd_config file:

[... snip ...]
AuthorizedKeysCommand /etc/ssh/ldap_keys.sh
AuthorizedKeysCommandUser nobody
[... snip ...]

Then, I installed the libpam-ldapd package (note the trailing d!) The nslcd daemon was configured like so:

# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldap://ad.myhouse.inthemiddleofthestreet.com:3268

# The search base that will be used for all queries.
base ***

# The LDAP protocol version to use.
ldap_version 3

# The DN to bind with for normal lookups.
binddn ***
bindpw ***

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com

# SSL options
#ssl off
#tls_reqcert never
tls_cacertfile /etc/ssl/certs/ca-certificates.crt

# The search scope.
scope sub

# Mappings for Active Directory
# This is the important bit; these fields match up with the fields added by Directory Services for UNIX
pagesize 1000
#referrals no
filter passwd (&(objectClass=person))
map    passwd uid              sAMAccountName
map    passwd homeDirectory    "/home/$sAMAccountName"
map    passwd loginShell       "/bin/bash"
map    passwd gecos            displayName
map    passwd uidNumber        objectSid:S-1-5-21-3623811015-3361044348-30300820
map    passwd gidNumber        objectSid:S-1-5-21-3623811015-3361044348-30300820
# If you wish to override the shell given by LDAP, uncomment the next line
#map    passwd loginShell       "/bin/bash"
filter shadow (&(objectClass=person))
map    shadow uid              sAMAccountName
map    shadow shadowLastChange pwdLastSet
filter group  (objectClass=group)
#map    group  gid              member

Note the attribute map above; the uidNumber and gidNumber maps are needed otherwise things will break when it tries to find your user in the default SimpleDirectory schema. Next, nsswitch.conf needs to be updated to use ldap:

[... snip ...]
passwd:         compat ldap
group:          compat ldap
shadow:         compat ldap
[... snip ...]

And, lastly, /etc/pam.d/common-session needs to be augmented for PAM to create the home directory on first login. Just stuff this line at the end:

session  required pam_mkhomedir.so skel=/etc/skel umask=0022

And that's what I did! I still need to sort out sudo functionality. Maybe that'll be a future update.

Software Builds at EA: Sourcing Code

In my previous post I discussed the general overview of EA's build framework, the one with the imaginative name Framework. Relegated to one of the last paragraphs was a mention of the package server and how it is used for enabling sharing of code across teams.

Since that post was written we've added a new mechanism to Framework to share code by means of specifying its location via a URI in the masterconfig file.

The URI sourcer has two back-ends currently implemented, one for fetching code from Perforce, and another for fetching from NuGet.

Perforce URI's encode the server, depot path, and the specific changelist or explicit head revision specifier of the software to pull down. This last condition is meant to improve build reproducability; anyone with a particular master configuration file will have the same code from which to build, local changes notwithstanding.

Because EA is a very distributed company, with studios located across several continents, and because these packages can be large (multi-gigabyte), Framework keeps an internal mapping of central Perforce servers to studio-specific proxy and edge servers, allowing users at these studios to avoid saturating slow trans-Atlantic and trans-Pacific network connections.

As Microsoft moves to distribute more components via NuGet we were looking to add support to Framework to support interoperation. By specifying a NuGet package name, version, and a URI, Framework will fetch the code from the NuGet mirrors. It then installs the package and wraps any contained DLLs in the Framework package scaffolding. This step is not strictly necessary, we could add the package to the MSVC packages.config and be done with it, but it adds a convenience layer to permit any package in the build hierarchy to express dependencies on the NuGet library as if it was a normal Framework package.

This is but the first step in enabling better distributed development workflows. Being able to pull down source code from someone else's server is handy; being able to fork a sub-component package, make a change, share it with your team and/or send a pull request back to the maintainer, while fitting in this same explicit versioning model that Framework relies on and using backing technologies that handle repositories in the tens or hundreds of gigabytes with no problem (trololol), is going to be a lot of fun. Or awesome. I'm hoping for awesome.

Software Builds at EA: The 5000' View

A couple of days ago this tweet by John Carmack popped up. In case the link ever goes away, he says, "Dealing with all the implicit state in the filesys and environment is probably going to be key to eventually making build systems not suck."

At EA we have spent a lot of time developing build systems that don't suck. They're not perfect but as we develop applications and games targeting platforms from Android to Xbox, plus everything in between, they work incredibly well for us.

Framework

The cornerstone of our build infrastructure is a version of NAnt that was forked eons ago and has undergone essentially a complete rewrite. Although the base functionality is still there, and still used, a lot of effort has been put into establishing it as a base for the higher level components.

NAnt is just one piece of a system called Framework. In addition to NAnt, Framework includes features for managing build configurations, SDKs, and software distribution. There are a few core concepts in Framework that I think have made it much easier to manage the build morass: modules, packages, dependencies, masterconfigs, build configuration management, SDK packages, and the package server.

Module

A module is Framework's representation of an artifact producing build process. At a minimum a module will have a type (static/dynamic library, executable, C# assembly, etc.), a name, and a set of input files. Modules have a number of optional components such as custom compile options, custom preprocessor definitions, custom warning suppression, sets of resource files/include files/object files/libraries, and dependencies.

Package

Packages are a collection of modules. Every package in the Framework world has a version, even if that version is "unversioned". A package contains a description of how to build its modules, its public exports (such as public include directories, build artifacts, etc.), as well as a manifest which allows the package author to include structured metadata that can be inspected by the package server, continuous build/test systems, etc.

Dependencies

Dependency handling is one of Framework's killer features. Dependencies can be made onto other modules in the same package, onto other packages, or onto other modules in other packages. For example I can specify that my module has a dependency on EASTL and the build system will ensure that I am using the correct version, build it if it is not already built, and set up the necessary include and library paths.

Masterconfig

Versioned packages lose their utility if there is no way to enforce the versions being used. The masterconfig file is the bill of materials for the application and specifies the version of each dependency, each SDK, and each configuration component.

One of NAnt's script concepts is the property. Essentially a global variable, properties are used for tweaking the state of the build system. Some are set in build files as crude variables, others like the particular configuration to build are set on the command line. A third way to set properties is to encode them in the masterconfig. This ensures that the whole development team is using the same Android SDK level, for example, as the SDK package may provide several.

Most build system state can be controlled in the masterconfig file(*), and checking the masterconfig file into source control can be a easy and effective way of ensuring the team is building with consistent settings.

(*) Provided that the build script writers do not start using environment variables on the side :-)

Build Configuration

Framework also provides global build configurations that describe how to perform a build. These build configurations take high level build goals (ie: optimized 32-bit Windows build) and use it to determine the required compiler invocation command lines.

These build configurations are used as the base for building a project and all of its dependencies. Although a particular dependency may override its build settings for warning suppression, custom optimization settings, etc., for the most part these are slight modifications made to the global configuration.

SDK Packages

SDK packages are not handled any differently by Framework than other packages, but Framework's versioning forces consistent use of SDK versions by all developers. Many SDKs are packaged in ready-to-go forms that the build system will download automatically and use to build, skipping the hassle of having to find the right SDK package and installing it. Or installing it and its numerous components.

Package Server

Where versioned packages come into their own is as a basis for sharing technology between development groups and teams. To make this sharing easier we run a server for people to post their packages on, not entirely unlike Freecode/Freshmeat. If Framework cannot find locally a package listed in the masterconfig it will try to fetch it from the package server.

Less Make, More CMake

Owing to its NAnt heritage Framework has the ability to perform builds itself but with Visual Studio support demanded by most developers it is being used more and more as just a front end for generating project files for Visual Studio or Xcode. 

French Immersion

About this time last year I was traveling from Vancouver to Montreal and eventually to London for work. It wasn't the first time I had done that particular route but it was the first time I drummed up the courage to dust off my awful western Canadian anglophone elective-course high school French and speak la belle langue wherever I went.

Getting into a cab at the airport I asked the driver to take me to my hotel at neuf-cent boulevard Rene Levesque. The driver asked me where I was from, where I was going to, what I was doing in Montreal. In French. I answered, haltingly at first, only slightly less haltingly as old words I had not used for years started to bubble to the surface, punctuating my sentences with apologies for my awful French. He told me his English was worse and encouraged me to practice. We mostly talked about hockey for the rest of the ride.

Searching out dinner I figured I'd keep the momentum I had been building and ordered in French. The waiter responded in English. Dammit.

Two days later I was on the red-eye from Montreal to London. A lady sat down next to me. I asked her in English if she was traveling beyond London. She shook her head apologetically. "No English".

Ou voyages-tu, apres Londres?

"Ah! Syrie!"

It could have been "A Syrie" but I could swear there was an exclamation in the middle.

This was well after the Syrian civil war reduced formerly great cities like Homs to dust and ghosts. She was traveling to see her family whom, she assured me, was neither dust nor ghosts. We talked for the whole plane trip. In French. I think she knew more English than she let on, but she wasn't comfortable speaking it, and I knew no Arabic. We mostly talked about the past, present, and future of Syria.

Time has taken from me many of the the details of our conversation. I do not know where she was traveling within Syria.

After arriving in London I wished her well on the next leg of her trip and thanked her for the opportunity to practice my French.

I hope she was able to get out with her family.

C and C++ need inline directives at the call site

I don't like the inline keyword, I think it's incredibly broken.

Inlining heuristics make reasonable assumptions about what should be inlined but there's basically no control by the user unless they use something like VC++'s __forceinline directive. Automatic inlining works well with trivial functions, the kinds that populate the cctype/ctype.h header, but fails when the desired inline function grows beyond a few statements.

The ability to inline code can be quite a powerful for developers, especially those who had to develop for consoles like the Xbox 360 and PS3, but because the compiler has incomplete information on where the inlining really needs to go and because there's no way in the language to give it better hints, there is often bloat in areas where you don't want it, and no inline expansion in areas where you do.

I've worked around this in the past by abusing macro expansion, but it's not a very clean solution, and makes debugging difficult.

What I would like to see would be an ability to use the inline declaration at the call site so that I can pick and choose where the inline expansion happens as needed. One example I can think of is multiplying two matrices in a video game. In some cases, say in creating the view/projection matrix, an inline expansion of the multiply would be unnecessary code bloat because this operation only typically happens once per game frame. 

Mat4x4 viewProjection = Mat4x4Multiply(view, projection);

While transforming joint hierarchies for animation there are typically a number of multiplications that happen in a tight loop where inlining would be beneficial:

for (int i = 0; i < numBonesInModel; ++i) {

    Mat4x4 transformedJoint = inline Mat4x4Multiply(jointMatrix[i], jointMatrix[parent[i]]);

}

I want more control over what gets inlined and what doesn't, without resorting to macros, and I think being able to specify at the call site what I want inlined would go a long way toward that goal.