Tooling Saves Your Bacon

Consultants have a bad reputation. It's common for development teams to view them with suspicion. They come in, make sweeping statements, wow your management team and leave as useless as they are dramatic. But at Vue School we think an engineering consultant is a different breed of consultant. Born in the trenches. A daily coder and a resource for you to lean on when having an external perspective can help your project ship.

Recently we helped a client with a consumer application sort through performance problems as they bring their application out into the world. Stability and performance are their priority. Their business goal is to prove the app in the market place, secure a round of funding, take the beta off and go like hell.

The CTO has been on the job for a month. He is comfortable with the product and can speak its goals. He is not sure of the code quality. What state is this app in? How does it perform? He wants to know what he is working with.

State of the application

There is a history of crashes while a user is broadcasting. Extensive efforts by the development team to address stability need to be verified. We need to concretely prove that broadcasting works. More than stability, broadcasting is not as performant as other services. We need to understand why? Is there something in Vue or the Firebase stack that is bottlenecking the app?
The feed is slow to load on mobile with an initial paint time above six seconds. Any aggressive scrolling drops the frame rate dramatically. On mobile there are reports of the application locking up when browsing back and forth between the feed and a video stream.

State of the nerds

The CTO's goal is to get a clear picture of the state of this application. Where are we? Are these bugs related? What part does Vue play? What happens when you broadcast for hours? He's a seasoned guy, having run many teams in different parts of the software industry. But he doesn't know this stack. Vue is new to him, Firebase is new to him.
He wants to get a baseline.

The lead engineer has been working night and day on this app for the past eighteen months. His was among the first commits to the project. He's been the architect and the force that's got the app built and working. He's gone hard and feels like this app is close, but he can't be the one to prove the app is in good enough shape to come out of beta.

He knows the app is working for him, but he's not confident about its stability for others. He likes his application architecture and design choices and isn't looking to make dramatic changes this far into the project.

My role is to man the lab-book.
I capture as much information as I can while testing the application. I collect copious numbers of screenshots documenting test runs with the HAR data that you can collect from Chrome’s dev tools. I would take about half an hour before each meeting to organize the collected data into stories I could tell about the days work.

The three of us get together Daily where I would present the facts in the lab-book (a google doc) to tell the story of the day’s work and let the CTO and lead engineer go as deep into the details of the work as they need and we'd arrive at my marching orders for the next day. Once a week we get all interested parties together for 45 minutes to go over progress and strategize next steps. We use Slack for regular focused conversations with the lead engineer.

Broadcasting a program

The customer support team makes regular and extensive product walk throughs. I was a power user by the end of the first day.

I had a localhost version of the app running within twenty minutes of getting access to the code. Any videos I created are stored on a publicly available 'dev' version of the site. I was immediately sharing videos of what I was doing with the team. I could go to the dev site from any machine and use it. This cleared the way for me to give useful and regular feedback.

Performance budget

Because of all the available tooling (Firebase shines here) I was able to explain to the team at the end of the first day what they are spending their performance budget on. I was able to explain and demo how each of their product decisions affected the application’s performance. You're done when you've got a problem down to business decisions. This was a great initial win.

From there it was about matching the performance of other broadcasting services. Digging deep into the webrtc implementation and understanding how Firebase and Vue are utilizing services.
Broadcasting involves WEB-RTC (Real Time Communication). I was careful to track every connection. It's the tedious work that matters the most. It's the primary value VueSchool is providing this first week of work. This makes the 'lab-book, our shared doc 📓 The most important artifact.

Beat it up

We ran a twelve hour broadcast with the goal of identifying any memory leaks and performance problems. I used the dev site directly from an older machine (2013 MacBook Air) to see how the application performs under stress and to verify that the application will work reliably for the widest possible pool of broadcasters.

We broadcast a program from one machine, added a second stream to the program from another laptop and then a third stream from a phone broadcaster to the stream. We ran three broadcasters on one stream for hours. We played pre-recorded videos on a loop. We also exercised the chat function and ran an ever changing audio singal through the broadcast.

I hit this poor machine as hard as I could for the twelve hour stream and it performed flawlessly. It streamed the full twelve hours and could have continued forever. The laptop was stressed the entire broadcast. We were able to determine that the encoder had a gate which capped itself at 60% CPU utilization, making reasonable tradeoffs between broadcast quality and machine capability. Memory usage was stable for a long period.

There were no leaks. I was able to prove to the team that their application was working as designed. One thing that impressed me about the CTO was how much he cared for his team. Though new, he was sensitive enough to pause a meeting and tell the lead engineer to take a moment and appreciate how reliable the app was. He praised his tech lead and encouraged him to feel good about the application’s performance.

We applied the rigor of our first few days to understanding how competing services use WEB-RTC. This gave the team context for how their product fit in the market.

Browsing a feed

With clarity around broadcasting we moved onto fixing performance problems on the feed.
The team was using scroll jacking to know when to load more resources. They had configured their player to load four broadcasts at a time and to cache a significant portion of each broadcast up front. This caused a massive resource dump every time the user scrolled the feed.

Intersection observer is supported across major browsers and is the perfect solution for loading large resources. Intersection observer is supported across major browsers and is the perfect solution for loading large resources. Therefore we abandoned the scroll jacking and opted for this more modern and appropriate solution. Then, with the application configured to begin loading before the resource came into view, the feed came to life. And this with minimal caching.

Intersection observer was new to the client so I took the lead in building the first solution. It was gratifying to see them adopt the technology with gusto and immediately apply the technique (as well as resize observer) throughout their codebase.

Using on mobile

The application is loaded on mobile via a WebView. An initial load of the feed could take upwards of ten seconds.

Digging into the build system I found that with every CSS change a new CSS file was created invalidating the existing CSS. The Fonts for the web application were embedded into the file making it about four megabytes. Every change to the styling of the app would cause this file to be reloaded.

Moving the fonts out of the CSS and modifying the built files into reasonable chunks solved the initial load problem.

Webpack config for optimized reusable static assets

  optimization: {
      splitChunks: {
        maxSize: 350000,
        minSize: 200000
      }
    }

Nasty safari bug

With the feed performing as designed we tackled a performance bug specific to WebkitView on mobile. CSS variables can cause screen repaints to consume the CPU and freeze the experience when switching between the web view feed and the native video player.

This is not a bug you can fix as a web developer. But we did address it. We Used stylus to find and replace all instances of var(--background) to stylus variables and rendered the application without CSS vars in production.

This change required no changes to the existing code and can be removed once the bug is fixed on safari's end. CSS var's are the future and we didn't want to give up on using them.

Another impressive trait of this development team was their willingness to commit resources to problems. Initially, I had no way to point the mobile app to my dev machine. So the turn around on testing solutions was convoluted. Within 24 hours of mentioning this to the CTO, I had the solution I needed to explore and solve the problem on mobile.

Takeaways

• Tooling is a hero; available by default with Firebase, the team had invested in it.
• CTO is available and willing to commit resources to remove roadblocks. He has clear priorities for me and though is new cares for his team
• Development team responds quickly to questions, are current and have an established culture of move fast and best practices.
• The team is thinking with me about how to solve performance problems. My role is to be diligent and provide the right level of information and offer solutions that make the smallest amount of changes to the codebase

At Vue School we have found that development teams with high standards value external validation and are motivated by a practical humility to have people review their work.