Shorebird Architecture

Shorebird is a set of tools that allow you to build and deploy new versions of your Flutter app directly to your users’ devices.

This document exists to explain how Shorebird works and to provide a high-level overview of the components that make up the Shorebird system.

Lifetime of a Shorebird Update

This assumes the developer already has installed Shorebird and has a working Flutter project, as covered in the Getting Started guide.

shorebird init tells Shorebird servers to create a new “App” entry associated with your account as well as writes the resulting app_id to shorebird.yaml in your project.

shorebird release builds your app using flutter build and then uploads the resulting binary to a private Google Cloud Storage bucket. The command also creates a “release” record in our database that associates the app_id with the release version.

shorebird preview can be used to download and run any of these privately stored release binaries on your local device.

Developers then typically take the resulting .aab or .ipa and distribute it to their users, typically via the Play Store or App Store.

At some point later, developers use shorebird patch to build a new version of their app. shorebird patch then downloads the previous release binary associated with that app_id and version number (patches use the same version number as the release they are patching) and then builds a “patch diff” from the combined release and patch binaries. This “patch diff” is then uploaded to a public Google Cloud Storage bucket where it will be served to devices requesting an update with the corresponding app_id and release version.

shorebird patch also creates a record in our database that associates the patch with the app_id and release version and alerts our “patch check” servers that a new patch is available. Patches can optionally be “staged” when uploaded. They will then not appear in patch checks until the patch “channel” is set to “stable” by the developer in the Shorebird Console.

Release binaries which were built with Shorebird contain Shorebird’s updater library. By default, the updater library will check for patches every time the app is started. This is done via a background thread to not slow down launch. The updater code makes a single request to our “patch check” servers which are again Google Cloud Run instances. The “patch check” servers respond with the URL of the patch diff if one is available.

A typical patch check is made via https, and contains:

{
  "app_id": "uuid...",
  "channel": "stable",
  "release_version": "1.0.0",
  "patch_number": 1,
  "platform": "android",
  "arch": "arm64-v8a"
}

A patch response contains:

{
  "patch_available": true,
  "patch": {
    "number": 1,
    "hash": "sha256...",
    "download_url": "google cloud storage url"
  }
}

The code to do these checks is open source as part of the Shorebird Updater library and may change from what is documented above: https://github.com/shorebirdtech/updater/blob/main/library/src/network.rs

The updater library then downloads the patch diff and applies it to the release binary. The patch diff is a binary diff. The updater library also checks the hash of the patch diff to confirm download integrity.

The hash is not meant to be a security feature, but rather a way to detect errors in the patch diff. A common error we see is that developers shorebird release with one source and then actually build and release a different binary, resulting then in invalid patches. This hash helps detect such errors. We currently do not sign patches, but plan to add that capability in the near future.

The modified Flutter engine also reports successful or failed launch of a patch back to our servers the next time it makes a patch check.

A patch event contains:

{
  "app_id": "uuid...",
  "arch": "arm64-v8a",
  "platform": "android",
  "type": "PatchInstallSuccess",
  "release_version": "1.0.0",
  "patch_number": 1
}

These events are used to display patch install analytics in the Shorebird Console.

If a patch fails to launch after install, in addition to sending a “PatchInstallFailure” event, the updater library will also mark that patch number as “bad” locally and refuse to boot the app with that patch number again. This is to prevent a bad patch from causing a crash loop on the device.

Shorebird Components and Source Code

Shorebird consists of 3 major parts:

1. The Shorebird CLI tool that you use to build and deploy your app from the command line.
2. A modified Flutter engine that is include in your app.
3. Our public-cloud infrastructure which hosts your app’s updates.

The Shorebird CLI

The shorebird_cli commands are documented on this site. Most of the build logic is just wrapping the flutter tool and it also adds a few commands to interface with Shorebird’s servers.

Flutter Modifications

Code push requires technical changes to the underlying Flutter engine. To make those changes required forking Flutter.

We had to fork 4 Flutter and Dart repositories to make Shorebird work:

flutter/buildroot “The Buildroot”

The flutter/buildroot repo contains the build scripts that are used to build the Flutter engine for various platforms. It’s separate from flutter/engine in order to share code and configuration with the Fuchsia build system.

We forked this repository to make several small modifications:

1. Expose our updater symbols from the Flutter engine up to package:shorebird_code_push

2. Include libunwind since the Shorebird Updater (Rust) code requires it for backtraces.

The Shorebird Updater is a Rust library which is linked into the Flutter engine via a static library (libupdater.a) that exposes a C API.

Because the default Flutter engine linking flags hide all symbols from linked static libraries, and because we need to be able to expose the shorebird\_\* symbols from libupdater.a to the Dart code, we made one change to buildroot and then a second change to the engine to place the symbols on the allow-list.

Note

You can see our buildroot changes on GitHub by comparing our trunk to the upstream buildroot, e.g. master: https://github.com/flutter/buildroot/compare/master…shorebirdtech:buildroot:shorebird/dev

flutter/engine “The Flutter Engine”

The flutter/engine repo contains C++ code that runs on the device. It is responsible for rendering the UI, handling input, and communicating with the host.

We forked this code to add the Shorebird Updater, which lets the Flutter engine load new code from Shorebird’s servers.

Note

Our Flutter Engine fork is public. You can see our engine changes on GitHub by comparing our trunk to the upstream Flutter engine, e.g. for 3.22.0: https://github.com/flutter/engine/compare/3.22.0…shorebirdtech:engine:shorebird/dev

flutter/flutter “The Flutter Framework”

The flutter/flutter repo contains the Dart code that runs on the device as well as the flutter tool that is used to build and run Flutter apps.

We forked this code to be able to deliver our modified Flutter engine to change the version of the engine that the flutter tool uses and to include the shorebird.yaml configuration as part of the asset bundle.

Note

Our Flutter Framework fork is public. You can see our engine changes on GitHub by comparing our trunk to the upstream Flutter Framework, e.g. for stable: https://github.com/flutter/flutter/compare/stable…shorebirdtech:flutter:shorebird/dev

dart-lang/sdk “The Dart SDK”

The dart-lang/sdk repo contains the Dart SDK, including the VM, dart2js, core libraries, and more.

We forked this code in order to teach Dart how to run modified (patched) code in an interpreter while being able to run all unmodified code on the CPU.

Note

Our Dart SDK fork is private currently. It will likely be public in the future and we will work to upstream many our changes as the team grows and the product matures.

Installing a Forked Flutter

When you install Shorebird, it installs Flutter and Dart from our fork. These are currently not exposed on the user’s path, rather private copies that Shorebird will use when building your app.

This was necessary to avoid conflicts with other Flutter installations on the user’s machine. Specifically, the way that Flutter downloads artifacts is based on the version of the engine. If we were to use the same version of the engine as the user’s Flutter installation, then we would overwrite the user’s engine artifacts.

We deliver our artifacts to this fork of Flutter with two ways. First is we change the version of the engine in the flutter tool. Second is we pass FLUTTER_STORAGE_BASE_URL set to download.shorebird.dev (instead of download.flutter.io) when calling our vended copy of the flutter tool.

Currently this means shorebird will not work in an environment where the user needs to use FLUTTER_STORAGE_BASE_URL to download Flutter artifacts from a private mirror (e.g. a corporate network or China). https://github.com/shorebirdtech/shorebird/issues/237

Serving Forked Binaries

We also use a custom server to handle requests from flutter for our modified engine. The source for that server is here: https://github.com/shorebirdtech/shorebird/tree/main/packages/artifact_proxy

The artifact proxy is hosted at https://download.shorebird.dev via Google Cloud Run.

This proxy knows how to serve the modified binaries from our Google Storage bucket, as well as how to forward along requests to Google’s Flutter storage bucket for unmodified binaries for all parts of Flutter we didn’t have to modify.

Shorebird’s Cloud Infrastructure

Shorebird’s public-cloud based infrastructure is responsible for hosting your app’s updates. It’s a set of services that handle the following:

• Release binary (private) storage
• Patch binary (public) storage and serving (via global CDN)
• Patch check requests (via Google Cloud Run)

We also provide a web-based console for developers to manage their apps and view analytics as well as the underlying database that powers the patch checks and console.

Currently all of our infrastructure is hosted on Google Cloud Platform although we will likely expand to other cloud providers over time. If we do, we will update our privacy policy to reflect such. All of these services are currently set to use US regions by default.