The BetterCloud Tech Stack
An inside look at the technologies and techniques used by the BetterCloud tech team to build the world’s first SaaS application management and security platform.
Before diving into our tech stack details, I wanted to share what it is we actually do with the technology we use. For those that aren’t sure what we do, BetterCloud helps companies manage and secure their SaaS apps, users, and data. We like to call it “mission control for IT.”
The application ingests data from multiple apps, displays information in a way no other IT application can, and lets you build cross-app automations to make IT operations easier and more reliable.
From an engineering perspective, creating the BetterCloud platform is an enormous undertaking and ongoing challenge, which is why we invest so much in our tech stack.
The 3 A’s of Building BetterCloud
Aggregation
We pull an enormous amount of external data from multiple SaaS applications into our system. This allows us to do some very interesting things in a single UI. For IT, this is a huge asset because they no longer have to jump from one native admin console to the next.
To further explain, BetterCloud’s aggregation capabilities let IT admins search for a single user and see ALL of their information from ALL of their connected SaaS applications in a single UI.
For example, I would be able to check that a recently promoted user’s title was changed in G Suite, Slack, and Zendesk; or when someone puts in their two weeks notice, an IT admin can check the user’s file permissions in G Suite, Dropbox, and Slack Channels.
An IT admin can take actions on any of the data that we sync internally. (e.g. She can add a user to a Slack channel or share a Dropbox file with another user.)
Automation
One of the most time-consuming tasks in IT is onboarding and offboarding users. This often consists of setting up or revoking accounts and permissions in multiple SaaS applications, adding or removing the user from groups or channels, and sharing or cutting access to files. The checklist might be 50+ line items long and the entire process could take hours to complete.
BetterCloud provides a platform for building custom workflows that can automate this process. A BetterCloud Workflow consist of three elements: WHEN, IF, and THEN.
WHEN a Google user is created, IF the user is in the “Atlanta/Engineering” org. unit, THEN take these specified actions.
The actions can reference the event context or hardcode values, making it possible to create workflows like:
When a user is created in G Suite, create a Slack and Dropbox account with the same email, add them to the “Staff” and “ATL” Google groups, share all HR documents in Dropbox and Google Drive, add the user to all general Slack channels, create a Zendesk ticket for IT to setup a laptop, and send an automated welcome email and Slack message to the created user.
Alerting
Some SaaS applications provide configurable alerts for admins, but their functionality varies widely. Because of BetterCloud’s aggregation strategy, we are able to provide a reliable set of alerts for each SaaS application, regardless of whether the native app provides similar functionality.
We can even apply complex rules that are unavailable in native admin consoles. BetterCloud admins can set thresholds, assign criticality, define who should get notified, and decide through which channel the alert should be sent, whether it’s SMS, email, or any app that connects via web hook.
Alerts can also trigger automations to kick off a BetterCloud Workflow:
For example:
=> Alert when a user publicly shares a Google Drive file with a title that contains, “Confidential.”
=> When “Confidential Alert” triggers, un-share all offending files, suspend all of the user’s accounts, and send an email to “security@yourcompany.com.”
Now, lets dive into the technology we use to make it all work.
Exploring the BetterCloud Tech Stack
Microservices
We use SpringBoot and Java 8 to implement most of our microservices. Spring provides the framework for REST API, security, messaging, data store layer, and a lot of other nice functionality. Most teams use Redis for distributed caching.
The Alerts teams use Scala and Flink (see BetterCloud at FlinkForward) for complex stream processing.
Datastores
We use MySQL (Percona) as the source of truth for relational data and Vitess to handle sharding larger data sets. Relational data is often denormalized and mirrored in ElasticSearch to support datagrids with complex filters.
Time series data is stored in Cassandra.
Secure data, including passwords and secret keys, is stored in Vault. BetterCloud open-sourced a Java Vault Driver.
Hadoop and Hive are used for long-term storage.
Communication
Microservices leverage Rest API’s for synchronous communication and Kafka 0.10.* for robust async messaging. We use Confluent.io’s flavor of Kafka and Avro for compression, schema enforcement, and performance.
BetterCloud open-sourced a debug utility for viewing and sending Avro messages via Kafka called Kadmin.
Rest APIs enforce authentication with a custom BetterCloud JWT for public endpoints and HMAC headers for internal endpoints.
UI
BetterCloud’s UI is built using Angular 1.4 with custom Material Design styles and components. Babel is used to transpile ES6 and Webpack is used for a user-friendly dev environment and optimized production builds.
Redux maintains the client applications central state, and Karma is used for testing client code. Nginx is used in our development environments as a reverse proxy to handle CORS, authentication, and module development.
Build And Deployment
All projects are built using Gradle as the main build tool and leverage Jenkins to manage continuous deployment. Gradle delegates front end builds to webpack and uses custom plugins to build JAR’s, WAR’s, and eventually Docker images.
All infrastructure and microservices are deployed to Google Cloud Compute as Docker images managed by Marathon and Mesos. Chef helps with build and configuring the VMs.
All REST requests are passed through a couple layers of load balancers and reverse proxies using Google Load Balancer, Nginx, and HAProxy.
What’s Next?
BetterCloud is investing in many technologies and frameworks that put configuration over code. As we scale to connect with more SaaS applications, we need a way to use existing code for new functionality.
An example of this pattern is Spring Cloud Data Flow project. Spring Cloud Data Flow provides the ability to build streaming pipelines from reusable and configurable components deployed as microservices.
