Ten Years of Brains: EdTech Lessons in the Decade After Rails for Zombies

A decade ago, my first week at Envy Labs was a bit hectic. September’s edition of the Orlando Ruby Users Group fell on the same week, and it would serve as the trial run for an idea we’d dubbed Rails for Zombies — a web-based course introducing software developers to Ruby on Rails. I know what you’re thinking and no, the zombie thing wasn’t as overdone back then.

Despite applying the interface design same day and fixing bugs well into the evening, the alpha application was a hit. Wider release saw the same success, and kicked off our four-year product journey in the form of Code School.

Ten years after our first foray, educational technology is still in its infancy. The platforms are more impressive, the companies behind them are much larger, and the market continues to grow, but the core problems haven’t really changed. As an industry, there are hurdles to hop around content creation, gamification, feedback, and a host of other experiential needs. Drawing from our own bootstrapped entry and the many third-party builds we’ve had a hand in since, let’s take a walk through the lessons still being learned.

Learn to Code, in a Browser

Rails for Zombies was born from a pair of issues:

• Setup. Certain coding languages, like Ruby, rely on a user’s machine to have the right combination of packages and configurations. When we’d run in-person workshops, most of the session would be spent tinkering with attendee’s laptops.
• Practice. There’s no substitute for learning to code by writing code, and the application’s killer feature was a series of challenges that did just that. Code School’s eventual tagline would be Learn by Doing — a motto shared by quite a few offerings these days.

After the alpha version’s successful meetup appearance, we gave the course a few rounds of polish and tossed it into the wild to see what would happen. What followed — with our team crowded into a room with live analytics and sandwiches — cemented the product vision moving forward.

Tackling that initial set of problems, though, was only the start. They gave us a solid entry and a unique take, but the real challenges behind technical training ran deeper.

The Hardest Part: Creating Quality Content

Even though our course ideas came from consulting experience, explaining something is wholly different than doing it. Teaching required learning.

It’s been unanimous from course authors (myself included) that serving up quality content is — by far — the hardest part of working in educational technology. Good content can overcome a bad interface, but no amount of experiential or presentational polish can save bad material. It’s been true for ourselves and for every training-based client we’ve had.

Within content creation, the hardest part (of the hardest part) is putting in the work to create solid technical challenges. Take Rails for Zombies, where the process looked like:

• Dream up a coding task that reinforces the material
• Adapt it into code and create any supporting assets
• Write conditional feedback for incorrect answers
• Thoroughly test the interactivity and edge cases
• Test via early access, find even more edge cases, head back two steps and start again

Running through all of that is time consuming. And taxing. And much harder than tossing in a multiple choice question before calling it a day. Newcomers to this variety of training are always excited by the potential behind complex, in-context testing, but that complexity also extends to the author. More than once, we’ve had to stage interventions with clients to ask why technical challenges were mysteriously absent from the second half of a course.

One final note on content: As it ages, it’s painfully similar to technical debt in software. Poor quality instruction, poor quality media, and any author shortcuts all live on until the course is removed or retired. Like a boat dragging its anchor, new versions of the interface will always struggle to bring the worst items in the catalog along for the ride.

The Uncanny Valley of Technical Training

You’re in a simulator that tests your ability to drive. Much to your annoyance, the brake pedal only works about once every three presses. And the blinker lever is reversed.

Content is king, but the interface behind technical training is pretty important too. Recreating experiences people use day in and day out — and may be an expert at already — is a balancing act of features and affordances. Trust is easily broken.

Our learn-to-code courses bypassed problems like machine setup, but the in-browser editor was always a work in progress. Users were asked to write code in an unfamiliar environment that lacked the key bindings, highlighting, and error checking that they were used to. Over time, those features would make an appearance, but only after reaching relative parity with their real-life counterparts.

A Subscription Model

Rails for Zombies’ success begot Code School, where we were able to expand the team and make all sorts of nervy decisions like monthly releases, subscription models, and themes that weren’t zombie-centric.

For the next several years, we continued to iterate on course delivery, tweaking interfaces for newcomers, seasoned developers, designers, and collaborations featuring GitHub, O’Reilly Media, and Google Drive. Trying new things each time made regular releases tricky, but the agile approach wound up working out: The EdTech industry was (and still is) learning about itself.

This next set of lessons comes from the product’s expanded audience and dozens of takes on the user experience.

Feedback is Hard to Collect, Help is Hard to Give

The core promise of technology-based learning is scale. Content becomes accessible to virtually anyone, on demand, and doesn’t rely on instructor availability.

With scale comes depersonalization. If someone runs into an issue, providing individualized assistance ranges from difficult to impossible. It also relies on the user seeking it out or being prompted at the right time. Everyone at Code School pitched in to help with support messages, but we’ll never know how many questions weren’t asked.

In this vacuum, getting quality feedback is also hard:

• Users don’t use the support lines to let you know when everything went right. Tracking data behind course completions and attempts helps here, but isn’t quite the same as hearing it out loud.
• The bad stuff is hard to swallow, but silent abandonment is far scarier. It doesn’t come with the opportunity to fix it, and the chance to fulfill the platform’s promise is gone.

Gamification is Morally Gray

Code School launched just before the heyday of all things gamification, and featured its own take on achievement badges and points. We thought they’d be a fun touch for completionists, and a neat thing to theme with each release.

Users absolutely ate it up. Way beyond what we expected.

Despite the Who’s Line is it Anyway?-style approach (“everything’s made up and the points don’t matter”), we were guaranteed a support ticket any time someone thought they’d unfairly lost points.

Gamification is better documented these days, and the benefits are hard to knock. It motivates, encourages routine, and fosters loyalty — all good things, when used for the right reasons.

As you’ve probably guessed from this section’s title, it’s not all clear skies. Loyal customers are easier to tap for microtransactions and future purchases, making them ideal targets for platforms that put the bottom line ahead of the educational mission. Even our best efforts had missteps:

• Public achievements are great, but not everyone is cool with publicizing what they’re learning. Especially if it’s remedial or several steps behind others.
• Hints can be helpful, but users will do whatever it takes (including asking for answers through support requests) if getting one means taking something away from them, like losing points.

Network Administration Certification

Zombies led to Code School, which led to (as a fortunate side effect) building similar platforms for other organizations. Things like Python in data science and network administration.

Branching out offered the best bits of experiential learning we could’ve asked for:

• Different sets of question types
• Interaction with real and virtualized hardware
• Users that weren’t peers in our field
• Larger teams of content creators
• The consequence of exams leading to real-world certification

Those new perspectives led to our last pair of recurring, root issues in the field.

The Classroom Experience is Tough to Recreate

It isn’t a problem new to 2020, but the intricacies of distance learning this year have only highlighted just how hard it is to recreate a classroom remotely.

We mentioned earlier that scale is a core benefit to educational technology. Some platforms, though, are willing to give up some of that accessibility to support live, instructor-led training. Timed examinations, individualized feedback, and special instructor tools (like unique changes to content, student progress tracking, and shared or mirrored challenges) all add development time for features that aren’t incredibly useful outside of a classroom-style context.

Professional certifications and their related prep classes seem to veer in this direction the most often, and they carry an air of added consequence. Application failures and missteps here aren’t just an annoyance: They can affect whether or not someone passes an exam necessary to their career.

Users Have Dramatically Different Goals

We were taken aback the first time a client asked for a shortcut to the next assessment.

Learning interfaces tend to be optimized for that bright-eyed newcomer with a fresh backpack and unused notebook. The kind of person that’ll read every word, pore over every image, and watch every minute of video we provide to learn something for the very first time. They’d never dream of bypassing the carefully-curated, linear path with a shortcut.

The reality is a bit more utilitarian. Users may be skimming for a quick refresher, chasing a continuing education checkmark, or constantly stopping and starting. Either way, they’re hoping to move through the content and be assessed as quickly as the platform will allow, and there’s absolutely nothing wrong with that.

Unless, of course, we’re talking about public safety training — like systems we’ve built for fire and EMS courses — with time and completion requirements that serve as a precursor to a different set of features.

It’s Still the Beginning for EdTech

The technology and resources behind educational technology keep getting more and more impressive. And centralized, too, as with our front row view when Pluralsight purchased Code School and began integrating interactive challenges into their course catalog.

While content delivery can only improve, the biggest leaps are still to come on the creation side. Building a solid, assessment-driven course remains hardest problem. And it only deepens with needs around feedback, instructor tools, and real world usage. Hopefully, after another ten years, we’ll have innovated our way to the next level of issues.