How we do mobile engineering at Venmo
Mobile Engineering has been slow to reach any kind of maturity.. so many apps begin their lives as ‘hacks’ that find success and are ‘maintained’ until the company is acquired. This is a contributing factor to why so many mobile apps that are acquired are sunsetted or completely rewritten — they prove to be unmaintainable and duck-taped together.
At least in part due to this, there aren’t many resources online about how mobile Engineering works – this is an attempt to outline the way that we approach mobile engineering at Venmo. It is by no means comprehensive and will likely be revised and added to over time, as well as followed up with posts outlining changes to the way we do things.. it is after all undergoing constant iteration and (hopefully) improvement. I hope that it’s helpful, if only to frame a conversation for how your own process works. ☺
This is a high level overview of some of the main aspects of mobile engineering. Each heading here could itself become a blog post (or chapter in a book *hint*) but I wanted to draw a line to get this post out — a brief overview of topology of mobile engineering
Our Goals
At Venmo, we’re in it for the long game. That means that we need a maintainable codebase which is easily extendable and flexible to ever-changing company goals. While our product is social, we’re also a finance app and so we have to take a very serious approach to software quality and security. Users feel safe using Venmo because they trust us — part of this trust is creating a stable & enjoyable experience.
We also want to be able to move as quickly as possible on new features and ideas to keep our market advantage. This means having highly modular code and good ability to leverage past work in reducing the difficulty of new feature implementation.
At a high level our goals for our mobile engineering process are to:
- Consistently ship high quality software
- Achieve a rapid (every fortnight-ish) delivery schedule
- Reduce the overhead of performing a release as much as possible
- Maximize code modularity and reusability
- Empower everyone on the team to ship, gather data and iterate on features they want to work on
Agile
We work with a loosely ‘agile’ process. Sprints are 2 weeks long and we aim to ship to the App Store at the end of each sprint. When features are going to take longer than that, they are split off and worked on concurrently with master which is always targeting this sprint’s release.
Week 1
- Monday: Sprint Planning Meeting
- Friday: (Potential) Internal Release (vX.Y.Za1)
Week 2
- Monday: Overview of bugs & feedback from Week 1 build
- Wednesday: Release Checkpoint. Everything should be merged — reconsider scope or release if not.
- Thursday: Team Internal Build (vX.Y.Zb1)
- Thursday: Release branch created: releases/vX.Y.Z
- Friday: Internal Release Candidate (vX.Y.Zrc1)
Week 3
(Concurrent with Week 1 of the next Sprint)
- Triage Bugs & Feedback from internal release (P1 issues — shipblockers — are patched)
- Release Candidate distributed to Friends & Family Beta group
- Submit to App Store. We always ‘Hold for developer release’ so that if any issues are discovered in the Beta group, we can pull the build.
Pull Requests, Code Reviews & Pairing
GitHub may be the greatest tool ever made for softare development. It makes revision control and code review / merging extremely simple. I’m going to assume you’re using it. ☺
Everything that makes its way in to master, begins its life on a branch and will be reviewed before merging. Merging is done in conjunction with the Release Manager & feature implementer. That way there is no confusion as to either: a. the status / merge-readiness of the work, or b. whether the feature should be included in the current release.
Say I’m going to extend Venmo to allow you to send or request money from multiple recipients at one time:
- I begin by creating a branch. Our branch naming convention is author_name/feature_description In this case: chris/pay_multiple_recipients
- Develop the feature (more on this later ☺ )
- Ensure that all tests pass (more on this later too!)
- Push my branch to remote origin
- Submit a Pull Request
- Ping the ios@ email list for review. We review for architecture, correctness, style (Style Guide later) and maintainability.
- Coordinate with the release manager to merge the Pull Request.
In cases where the feature implementation requires ‘architectural’ changes to the application, it’s often helpful to have two people working together on the feature in a pair. This means that once a general approach is decided, one can be focussing on implementation while the other continues to reevaluate the implementation from a higher level, triggering changes in approach as necessary. In these cases we still submit the feature for team-wide code review as there are some things which you simply won’t spot while in the implementers mindset of ‘getting it done’.
Testing & Quality
There are several main components to our quality strategy:
- Unit Tests
- Automation Tests
- CI / CT on Jenkins
- Manual Verification
- Dogfooding
- Production Crash Reporting
Unit Tests
For new feature development, where it makes sense we practice Test Driven Development (TDD). TDD is a great tool to ensure that you’re practicing YAGNI while writing maintainable code — TDD ensures that you write testable code that solves the problem as simply as possible. That said, the cases where TDD doesn’t make sense are just as important to consider, I would recommend testing the happy flow to drive development and then consider edge cases afterwards. 37 Signals wrote a great summary of how to know if you’re doing TDD wrong.
We use Specta as our testing framework and OCMockito for mocking and stubbing.
describe(@"VTAnalyticsSimpleTransmissionStrategy", ^{
it(@"transmits given events", ^{
VTNetworking *mockNetworking = MKTMock([VTNetworking class]);
id transmissionStrategy = [[VTAnalyticsSimpleTransmission alloc] initWithNetworking:mockNetworking];
NSString *testEvent = @"Some Test Event Name";
VTAnalyticsEvent *event = MKTMock([VTAnalyticsEvent class]);
[MKTGiven([event key]) willReturn:testEvent];
VTFSNConnection *mockConnection = MKTMock([VTFSNConnection class]);
[MKTGiven([mockNetworking requestWithAPIPath:(id)HC_anything()
method:VTFSNRequestMethodGET
parameters:(id)HC_anything()
completionBlock:nil])
willReturn:mockConnection];
[transmissionStrategy transmitAnalyticsEventsAsync:@[event] withSuccess:nil failure:nil];
// The network call is created
[MKTVerify(mockNetworking) requestWithAPIPath:@"/analytics/event"
method:VTFSNRequestMethodPOST
parameters:(id)HC_hasEntry(@"event_name", testEvent)
completionBlock:nil];
EXP_expect([mockNetworking requestWithAPIPath:@"/analytics/event"
method:VTFSNRequestMethodPOST
parameters:HC_hasEntry(@"event_name", testEvent)
completionBlock:nil]).to.beIdenticalTo(mockConnection);
});
});
Testing is focussed on model and state transitions. View-based functionality is verified predominantly using Test Automation as unit testing this is a mixture of difficult, pointless and time consuming. A terrible intersection.
Test Automation
Test Automation is scripting interactions with a running instance of your app to simulate a user actually using the app. It does this by simulating taps and swipes — a great tool for this (after investigating all of the tools available, the one that we use) is KIF (it’s actually maintained by Square ✌ .)
We verify all basic user flows using automation e.g.
- Signing up (including edge cases)
- Logging In
- Paying and Charging
- Cashing Out
- Adding and removing cards
In testing these flows, we also verify all intuitive functionality is intact, such as the side drawer working and buttons all doing what they should. This is part of the reason I don’t subscribe to the idea of verifying that ViewControllers are visible after button presses in test code.
Automation tests are broken in to two targets — ‘Smoke’ and ‘Regression’. The ‘Smoke’ tests execute in around 5 minutes and are designed to be run locally. The ‘Regression’ tests are a much more exhaustive suite of edge cases and stress testing which take around 30 minutes to execute.
CI / CT using Jenkins
As I mentioned, some of our tests we run locally before pushing and pull-requesting, others take a while and are run automatically on commit by our test server — a Mac Mini running Jenkins. If you don’t want to invest in your own hardware (which is awesome because you can run on actual devices)… Travis also does a good job of this as Cloud service.
Jenkins watches watches master and on commit will:
- Pull the latest commit on master
- Compile & static analyze (watch warning trends)
- Run unit tests
- Run Smoke Automation tests
- Run Regression Automation tests
- Create a ‘HockeyApp’ build
- Email us if anything went wrong
Test automation gives us a high level of confidence that the latest master is always stable & ready to begin a release process.
Manual Verification
No matter how many tests & how much automation you have in place, consider the following conversation:
Developer: We have a bad build in production
CEO: Did you even open the app before submitting it?
Developer No.
Who’s going to feel sorry for that guy? You simply can’t get around actually using the app before releasing it. We have a manual QA spreadsheet which used to be exhaustive and over time, many of the test cases have been marked as ‘Verified in Automation’. We still test upgrading from the current App Store version (never don’t test this!) and basic flows manually on a variety of devices.
Dogfooding
Venmo employees love Venmo and use it multiple times most days! That means that we have an active, willing and available group of test users. We ship internally every week with extended logging and debugging as well as crash reporting. This is for both quantitative and qualitative feedback. We have a very collaborative product development process and feedback from people in the company routinely gets worked in to the same or next version of the app.
Once a build has been verified as stable in dogfooding, it’s ‘promoted’ to be distributed to a group of ‘Friends and Family’. This is several hundred users who have been referred by employees and a few power users — they get early access to new features and help us to verify them in the wild.
People using the app is one of the best way of verifying that they will work in production — take advantage of the users you have inside of your company.. they’ll be excited to get early access to new features you’re working on.
HockeyApp is a great tool for distributing builds & if you have an Apple Enterprise Certificate, you won’t need to mess around with UDIDs and custom Provision Profiles for distributing within your company.
Production Crash Reporting
Crashlytics — use it. Crashlytics provides fully de-symbolicated crash reports from your app in production. It allows you to see where your app is crashing & how many users are being affected. We use this number and our Flurry sessions count to build a ‘% of sessions that crash’ metric — this is our ‘picture of quality’.
Release Management
We have a very flat structure at Venmo & while I have responsibility for mobile software quality generally, we distribute the tasks of managing releases around the team. Each release has a ‘release manager’ who takes on the ‘overhead’ of performing the release such as creating dogfooding builds, coordinating pull requests, verifying the state of QA and submitting to the App Store. This ensures that we have well-distributed knowledge on the process of release and allows everybody to take ownership of a release — including announcing it publicly on our blog.
The release manager is chosen at the beginning of the sprint, often it’s the person (or one of the people) who are working on the ‘banner feature’ of the release. We really like how much ownership that gives each member of the team on their release.
User Testing
We decide what to make and users just get what they’re given, right? Wrong.
Developing software is one of the most expensive things that most product companies do and so we owe it to the company (ourselves) to not bear that cost unduly. One way to reduce this risk is to do what we can to ensure that users will understand / use / be helped / whatever.. by the feature, before we actually develop it by conducting user testing.
User testing really warrants a post all of its own but the general idea is to bring a few new / existing users (depending on what you want to test) and have them interact with some form of prototype with the aim of completing discreet tasks while verbalizing their thought process. This prototype can be anything ranging from a paper prototype (Prototyping on Paper is an excellent tool to turn sketches in to an interactive on-phone experience) to a quickly hacked-together app which demonstrates the concept being tested, to a candidate for release with the feature. In general, the less tied to a feature / design you are when testing it, the more effective user testing will be.
If you’re testing on a real app (either a modification of your App Store app, or a separate demo app), Lookback is an amazing tool to be able to record both the iPhone screen and the testers face (using the front facing camera) by simply including a framework in the build. This dramatically reduces the friction of having to have a separate video camera / take notes manually. Testers are usually very comfortable with being recorded in this sort of situation. We use HockeyApp to distribute and install these builds as-well as our internal builds.
As with all feedback mechanisms — be prepared to listen & take action. Data is useless if you won’t act on outcomes.
Rapid Releases
We believe in releasing rapidly for a number of reasons:
- Test outcomes and experimentation can be iterated on more quickly. We do have some ability for remotely defined tests but they are limited in scope. This is a key area for improvement that we’re working on — Facebook are doing a lot of great work here.
- Bugs / issues that don’t warrant a hotfix release are resolved more quickly
- Engineers can get their features in to production more quickly ☺ — happy engineers are the best thing in the world.
- Frequent updates are rumored to be good for App Store rankings (who knows?!)
- Users know that we’re committed to providing them with the best experience and support possible. For an app with day-to-day utility, even an issue that’s only around for days-to-weeks can be extremely annoying. Months without support will likely cause the user to migrate from being a user and fan, to a public detractor.
At any one time, there is between 1 and 3 releases being worked on. The ‘current’ release (i.e. the release for this sprint) is a given — this usually where most people will be spending their time. There are also two other releases that could be under way:
- A hotfix. If a release has issues that are discovered in production which would have been considered a P1 (ship-blocking) issue in testing, we will issue a hotfix release. This will be branched off of the release branch from the previous version and be a 0.0.1 increment. e.g. a hotfix on v4.4.0 would be branched off of releases/v4.4.0 and named v4.4.1. This will be the minimum change-set possible to address only the P1 issue and will undergo the same testing as the main release.
- A long-running feature release. There are some things you simply can’t build in two weeks. In these cases, they will be developed in a feature development branch (e.g. our side drawer release took a couple sprints and was maintained in feature/sliding_drawer_redesign.) master is merged regularly in to feature branches to prevent large merge conflicts & potential bug introduction later. Features can span two sprints (on odd occasions three) but after that, they should be broken down in to staged work, ideally that can fit in to one sprint.
Code Reuse
We write code most of most days. Some of that code tests other code, making the code that we’ve written even more valuable because it often has 1-2x the amount of code, testing that code. Code is really expensive but it does awesome stuff.
Lots of code we write is basically the same code, put to slightly different purposes — that’s why high level languages exist after all and why iterating through data structures and converting between object types are all one-line operations. They’ve identified and abstracted out common functionality and included it in the language. We can do the same things in our applications, especially if you maintain multiple codebases within your team (in our case Venmo & the VenmoTouch client library.. and who knows what else in the future).
CocoaPods is a must-have tool for managing dependencies and versioning of vendor & open source code that you use in your application. If you’re not familiar with it, check it out. CocoaControls is a really awesome site for finding third party libraries which often achieve exactly what you’re looking for but are under community support (be sure to check out code quality and issues before blindly integrating!)
There are a couple of features of CocoaPods which not many people take advantage of:
- You can create and maintain a private PodSpec repo – this means that you can distribute and share internally, treating them as opensource projects, with semantic versioning and dependency management without opening the source up to the world (or anybody else even knowing they exist).
- You can use a local PodSpec to point at a folder within your folder structure to allow developing a pod in-line with your project while still maintaining it’s isolation as a standalone library.
Where possible, we create generic controls and libraries and include them in our Venmo PodSpecs repo on GitHub Enterprise. This has three main advantages:
- We can easily share code between Venmo & VenmoTouch where applicable
- Updates are shared as they are published but versioning is controlled granularly in the case of breaking changes to the interface.
- You simply think differently about ‘shipping a library’ than making a change to a file in your repository. You create it more generically and more attentive to documentation, interace cleanliness and quality in general. There are some things that the ‘open source’ model does very well — this is one of them.
A few examples of things we’re in the process of doing this with:
- Our ‘Add Card’ control (with CVV / card number validation etc..)
- A generic ‘Venmo’ modal view for all applications
- Our A/B testing and remote configuration framework
What to work on
The way that I think about this is that there are two factors:
- Business objectives — things that need to get done due to strategic priorities for the company
- What people on the team want to work on. This matters, a lot.
Sometimes there are things that just need to get done and you try to get excited about it, find the people most in to it and have them work on it. That’s okay, it’s business & life. But in the absence of these, I genuinely believe that the right way to prioritize feature development is based on what somebody is burning to get coding on. People do their best work & write their best code when they love what they’re working on. We all love Venmo, but there are some things that each of us are just super excited about. For me — that’s refactoring our codebase to have a clear client-library and application division. For Ayaka, it was implementing the new recents bar.. from conception to shipping in around two weeks because she wanted it & crushed it! For another engineer it was validating form input using Reactive Cocoa (check it out!) — boom, done.
This is a very high-level snapshot of the constantly evolving processes that we’ve built at Venmo to keep us shipping high-quality fast-moving mobile software. I’d love to hear any comments & feedback, and will keep adding to this and fleshing out topics over time.
Let's talk about it.
Head on over to Twitter and let's talk more.
Find me at @chrismaddern.