This is the story of the time I wrote some code, deployed it to production, and ended up bricking the server it was running on by frying the kernel.
This post is about perils of concurrency and race conditions. My code was nearly correct, but ultimately, there were two major synchronization bugs that killed it.
TL;DR: Can we use GPUs to get 10x performance/dollar when fuzzing embedded software in the cloud? Based on our preliminary work, we think the answer is yes!
Fuzzing is a software testing technique that supplies programs with many randomized inputs in an attempt to cause unexpected behavior. It’s an important, industry-standard technique responsible for the discovery of many security vulnerabilities and the prevention of many more. However, fuzzing well takes time, and fuzzing embedded software presents additional challenges.
Embedded platforms are not designed for the instrumentation and high-throughput computation required to find bugs via fuzzing. Without access to source code, practical fuzzing of such platforms requires an emulator, which is slow, or many physical devices, which are often impractical.
Most fuzzing approaches have used conventional CPU architectures or emulators, but we decided to use other commodity hardware to tackle this problem—in particular, GPUs. The recent boom in machine learning has driven down the off-peak price of GPUs and made massive GPU computing capacity readily available from all major cloud providers. GPUs are very good at executing tasks in parallel, and fuzzing is an easily parallelizable problem.
In this blog post, I’ll walk you through the design and implementation of this massively parallel GPU-based fuzzer. So far, we’ve implemented an execution engine that can achieve 5x more executions/second/dollar than libFuzzer—and there’s much more room for optimization.
Writing quality software is hard. Sometimes, software breaks in entertaining ways. However, when software runs everything from personal assistants like Alexa and Google Home to banking to elections, some bugs can be much more severe.
This past quarter, Armin Namavari and I tried teaching a class about how to write software that sucks just a little less. We focused on common problems in computer systems caused by certain kinds of silly (but very serious) mistakes, such as issues of memory safety and thread safety. The core theme of the class was, What are common problems with systems programming right now? How are people responding to those issues? How do those measures fall short? We wanted students to be aware of problems that have plagued the industry for decades, and we wanted to teach students how to use tools and mental models that people have developed to combat those issues. However, these tools are imperfect, and we also wanted students to experience and understand the limitations of such tools to be better aware of what to watch out for when building systems.
Last fall, I had the opportunity to serve as the head TA for a new introductory CS course at Stanford. As part of this, I worked with the teaching team on designing a new final assignment that gives students a taste of full-stack web development by having them build a miniature social media platform.
This was a fairly challenging undertaking. Most students were taking this class as a first formal CS course, so they had no experience in working with complex codebases. Furthermore, this was not a “how to do web programming” class. It was a programming fundamentals class that introduced web programming at the very end to help students contextualize their learning. As such, it was a significant challenge to walk students through the process of building a web application while ensuring they understood all the moving pieces and felt rewarded for their work.
This project was developed by Isaiah Bush and myself. Many thanks to Isaiah for his excellent work in building out the frontend portion of this project.
Learning multiprocessing is hard. It’s like landing in a foreign country where you speak the language, but the social conventions are wildly unfamiliar. All the words seem to make sense, yet despite your best intentions, you manage to offend every person you meet, and everyone refuses to cooperate with you. Worse, no one will explain why they are upset; at best, they mutter cryptic phrases like “EPERM” and “EINVAL”, and at worst, they glare at you and say nothing. You continue your march of faux pas until the kernel eventually banishes you with a segfault.
We wanted to make multiprocessing easier to learn through use of diagrams, providing transparency into what is happening in the kernel instead of making you fumble until you figure it out.
[This post is part of an effort this year to be more open about things I struggle with. In this post, I talk about my struggle with trying to be there for people in a meaningful way.]
My friend Arjun committed suicide last September. I’m “over it” in as much of a functional sense as possible, but I still think about him all the time.
I miss him so much. He was among my best friends in high school. We cofounded our startup together. When we left that to go to college, I would crash in his apartment when I came to visit. We played music together; he played drums and cajon, but he gave away his cajon to someone he said “needed it more.” He was always the first to experiment with new technologies and programming languages, and he always had a refreshingly grounded perspective of Bay Area startups. We shared the same cynicism for the recent explosion in entrepreneurship, but also the same passion for doing and building. I wish I could tell him about all the things I’m up to these days, brainstorm things for me to pursue after graduation, and ask for his advice. That will never happen again.
But here’s the funny thing: I don’t know if I would be so eager to talk to him if he weren’t dead. Death has an interesting way of doing that. He had moved to the Bay Area for work, but even though he was only an hour or two away, I only saw him a handful of times per year. In terms of frequency of contact, nothing has changed that much, but the finality of it feels so heavy. Everything has changed, forever.
I usually avoid posting publicly about life details. Most of the time, I don’t feel they’re worth reading about, and, being a perfectionist, I always spend too much time agonizing over how people might take in what I publish. However, I recently read Norah Borus’s blog about her life at Stanford, and it made me think about how much I appreciate other people’s writing. I like reading about what other people are up to; more interesting than seeing what people have done is hearing about what motivated them to do it and what they encountered along the way.
Even more refreshing, however, is reading about what people struggle with. The term “duck syndrome” is popular at Stanford, referring to the appearance that everyone is gliding effortlessly on the surface of a lake, but underneath, everyone is paddling frantically. When I look around, people seem to be doing amazingly well in life without skipping a beat, yet I know most people struggle, often with the same things. That’s why it’s so refreshing to read a blog like Norah’s. I like to think there is more to us than what we accomplish, that the ways in which we struggle and figure life out are worth talking about.
Of course, writing publicly is easier said than done. Many times, it is neither practical nor acceptable to talk publicly about what I’m struggling with. Regardless, I decided that this year, I want to try to do more of it. I haven’t decided how much I want to share – this post will be light – but as we kick off 2018, I figured publishing my end-of-year reflections would be an apt place to start.
I took Stanford’s CS 148 (Computer Graphics) this quarter. For the course’s final project, we were asked to design a scene and assemble/render it in a raytracer. The following is the image that I created:
I spent about a week on this project. While there is much I could have improved, I am pretty proud of the results, so I wanted to document my process and reflect a little on what I could have done differently.
I have been working on a tool that collects information about network requests to help debug failing integration tests. The tests click around a website using Selenium to make sure everything works as expected; if a test fails, network logs might help track down issues with API calls. Wireshark is helpful for analyzing HTTP requests over SSL/TLS, but I needed to figure out how to do this programmatically. Tshark is the command-line cousin of Wireshark (“terminal-shark”); it is quite a capable tool, but it took me a while to figure out how to use it for what I wanted to do.
Stanford provides a cluster of computers for use in CS classes known as myth that is equipped with gcc, gdb, Valgrind, etc. I’m using these computers as part of CS 107 this quarter. (If you aren’t a Stanford student taking CS 107, this post might be totally irrelevant to you, but maybe you can still get something out of it.) These machines can be a bit of a pain to use for someone who has never worked over SSH before, but there are a few things you can do to make life happier.
EDIT 6/8/20: This blog post is a bit out of date. I have written up some more recent tips here.
I recently bought a used Kinesis Advantage from EBay. Unfortunately, the keyboard came with a dead tilde key. I emailed Kinesis Support, who told me the following:
1). You can send us the keyboard and we can repair it for you. No cost if the keyboard is under warranty. Outside of warranty $40 + return shipping.
2). Since the problem is isolated to a single key on the left side of the keyboard, then we can send you a replacement right keywell (right side of keyboard with alpha/numeric keys) for $30 + shipping (or free if under warranty) and you can swap it out yourself . Only a screwdriver is required and step-by-step instructions are provided.
3). If you’re handy with a soldering iron, I can send you a few replacement key switches free of cost and you can swap out the bad key yourself, which as far as soldering goes is fairly simple, however I obviously wouldn’t recommend this if you don’t have experience soldering.
Since the keyboard was no longer under warranty, I figured I’d give the third option a shot. Kinesis mailed me three switches for free (which was very surprising and very nice), and I crossed my fingers and hoped I wouldn’t break anything. The repair turned out to be relatively straightforward, but because I couldn’t find much online about other peoples’ experiences doing this, I thought I’d post pictures of what I did.
When I reached the end of my senior year at the Illinois Math and Science Academy (IMSA), I had a difficult decision to make: Which college should I go to? I hadn’t been accepted to the schools I most wanted to attend, and most of the schools that did accept me costed upwards of $30,000 a year. My parents encouraged me to consider community college, but I struggled with that idea. I had never heard of anyone from my school ever attending a community college, and I thought I was above that. All of my friends are heading off to all these big-name, top-ten colleges, and I’ll be off to… community college? In the end, though, that’s what I decided to do, and I’m glad I went that route. I wanted to talk about my experiences for anyone that might be in a similar position.
I write a lot. I rarely publicly share what I write, but I write a lot anyways, and I consider it one of my most important habits. However, I know many people don’t write, and many who want to write don’t know what to write about. This blog post intends to change that!
I recently taught an introductory frontend web development workshop for the Harper Society of Engineers, and I promised I would post a writeup of what I taught. This is that writeup!
Far too often, I sit down in front of the computer to get some work done, only to look up and realize that eight hours have elapsed, I haven’t eaten, and my prescription has increased by 2 diopters. Break timers are handy, but many cost money and/or are too bulky and annoying for my preferences (they’ll pop up a big overlay on your screen saying HEEEEYYY RYAN TAKE A BREAK, which kills my concentration if I am thinking about a hard problem). My experience with most break timers is that I disable them within a few hours of using them. All I want is a subtle reminder, and I figure it shouldn’t be too hard to script that myself.
This semester, I am taking a class on assembly programming using Kip Irvine’s book “Assembly Language for Intel-Based Computers.” Unfortunately, the book should really be called “Assembly Language for Intel-Based Windows Computers,” as it is written for Microsoft Macro Assembler (MASM). It mentions that the programs in the book could be converted to TASM assembly fairly easily, but they will not run out of the box.
I could run MASM on a Windows virtual machine, but that would be fairly heavy to have running whenever I want to work on assembly. Instead, I wanted to see if I could run MASM with little overhead using Wine. It turns out that this is fairly easy to do!
I have decided to start a blog.
I have long considered starting one, but I never really knew what to write about. It often feels like every problem in tech has already been adequately explained on some other fellow’s blog in words better than I can manage.
subscribe via RSS