A Six-Year Android SDK Breakage Forced One Team to Rewrite Its CI Test Harness

In early 2025, a mobile development team of about a dozen engineers watched its CI pipeline grind to a halt. Builds that had passed for six years suddenly failed. The culprit: a single Android API method, deprecated since SDK 28, finally removed from the platform in SDK 34. The team had not noticed the deprecation warnings because their Gradle script suppressed them. What followed was a four-month rewrite of their entire test harness, a process that revealed how fragile Android CI can become when a single engineer holds the keys.

The SDK That Stopped Compiling

The team's Android app targeted a minimum SDK of 26 and compiled against SDK 33 for years. Their CI pipeline used a custom Gradle plugin that invoked ActivityManager.getRunningTasks to detect whether certain background processes were alive during integration tests. That method had been deprecated since API level 21, but Google did not remove it until SDK 34. When the team finally updated their compile SDK to 34, the method was gone.

The failure was not immediate. The first sign came when a developer pulled the latest SDK components and ran a local build. The compilation failed with a “cannot find symbol” error. The team's CI pipeline, configured to use SDK 33, still passed. But a separate branch that targeted SDK 34 as a forward-compatibility test broke. No one had run that branch in months.

Google's compatibility checker, part of the Android Studio toolchain, had flagged the deprecated method in earlier SDK versions. But the team had disabled those checks in their Gradle configuration to reduce noise. The warnings were invisible. As one engineer later put it, “We optimized for a quiet build log and paid for it with a silent time bomb.” This pattern is not uncommon: many teams suppress lint warnings to speed up builds or reduce cognitive load, but doing so removes a critical early-warning system. A similar dynamic occurred at another company where a team disabled deprecation checks for a Kotlin library and later discovered that a core extension function had been removed, causing a cascade of runtime crashes in production.

The removal was documented in the Android SDK release notes, but the team had not reviewed them. Their CI pipeline did not include a step that compared new SDK deprecations against the codebase. The six-year gap between deprecation and removal lulled them into assuming the API was stable. However, this assumption is dangerous: Google has removed several long-deprecated APIs in recent SDK versions, including AsyncTask (deprecated in API 30, removed in API 33) and LocalBroadcastManager (deprecated in API 28, still present but discouraged). The team's experience underscores the need for proactive monitoring rather than reactive fixes.

Why a Test Harness Became the Bottleneck

The test harness was a monolithic Gradle script that had grown organically over five years. It handled emulator provisioning, test execution, artifact collection, and reporting. The script was 2,800 lines long, with no unit tests. One engineer, who had since moved to a different team, understood most of it. The rest of the team treated it as a black box that “just worked.”

Each emulator boot consumed 4 to 8 minutes, depending on the API level and device configuration. The harness ran tests in parallel across four emulators, but the parallelization was fragile. A single flaky test could cause a cascade of failures that masked other issues. The team often spent hours re-running builds to identify which tests actually failed. This is a classic symptom of a test harness that has grown beyond its maintainers' understanding—a problem that affects many teams. For instance, a similar incident occurred at a fintech company where a Gradle script for running UI tests on multiple devices had a race condition that caused test results to be attributed to the wrong device, leading to weeks of false positives.

The bus factor was a known risk. The original author had left detailed comments, but they described what the code did, not why it did it that way. When the deprecated API broke, no one could safely modify the script. The team considered patching the harness to use the replacement API, but the mocking framework relied on internal Android classes that were also deprecated.

“We had a choice,” the tech lead recalled. “Patch the harness with a workaround that might break again in SDK 35, or rewrite it properly. We chose the latter, knowing it would take months.” This decision was not made lightly: the team evaluated the cost of a temporary fix against the long-term maintenance burden. A patch would have taken roughly two weeks but would have introduced technical debt that would likely require another rewrite within a year. The rewrite, while expensive, offered a cleaner architecture and better testability.

The Anatomy of the Breakage

The deprecated ActivityManager.getRunningTasks method returned a list of tasks that the system was currently running. The team used it to verify that their app's background service was alive after a user navigated away. The replacement, UsageStatsManager, required the PACKAGE_USAGE_STATS permission, which is a special permission that users must manually grant in Settings. That changed the test setup entirely. Moreover, UsageStatsManager does not provide the same granularity: it aggregates usage statistics over time intervals rather than giving a real-time snapshot of running tasks. The team had to adjust their test expectations accordingly.

The mocking framework, built on top of android.test.mock, used internal Android classes like ActivityManagerNative to intercept system calls. Those classes were hidden in SDK 34. The team had two options: use the public InstrumentationRegistry and UiAutomation APIs, or switch to a different mocking strategy. They chose the former, but it required rewriting every test that depended on the old mocking layer. This is a common pain point: internal APIs are often used by test frameworks to simulate system behavior, but they can vanish without notice. For example, the SystemProperties class, used by many test harnesses to override device settings, was hidden in SDK 29, breaking countless test suites.

There was no public alternative for UI-less task detection. The team had to rethink what they were testing. Instead of verifying that a specific task existed, they switched to checking that the service's process was alive using ActivityManager.getRunningAppProcesses, which was still available but less precise. The trade-off was acceptable: they lost some diagnostic detail but gained a maintainable test. However, this change also meant that the tests could no longer distinguish between the service running in the foreground versus the background, which reduced coverage for certain edge cases. The team documented this limitation and accepted it as a known gap.

The breakage exposed a deeper problem: the team had no integration tests for the harness itself. They could not verify that a fix worked without deploying it to CI and waiting for a full build. That feedback loop took 45 minutes. “We were debugging in production,” one engineer said. To mitigate this, the team later added a local test mode that allowed running a subset of tests against a real device connected via USB, cutting the feedback loop to under 10 minutes. But that improvement came only after the rewrite was complete.

Rewriting Without a Safety Net

The rewrite began with a migration from Gradle to Bazel. The team chose Bazel for its hermetic builds and cacheable test results. Bazel allowed them to define test targets that explicitly declared their dependencies, so a deprecation in an Android SDK class would cause a compile error at the target level, not a cryptic runtime failure. Additionally, Bazel's incremental builds reduced the average CI time from 45 minutes to roughly 20 minutes for most changes, though full rebuilds still took around 30 minutes.

The migration took three months. The team rewrote the test harness from scratch, this time with unit tests for every component. They adopted Firebase Test Lab for instrumented tests, replacing the local emulator farm. Firebase Test Lab provided real devices at a cost of roughly US$ 2,000 to $3,000 per month, depending on usage. That was more expensive than the old emulator setup, which ran on spare on-premise machines, but the reliability was higher. The team calculated that the old emulator farm had a hardware failure rate of about 5% per month, causing intermittent test failures that wasted engineer time. Firebase Test Lab's device availability was guaranteed, and the team no longer needed to maintain the physical machines.

The team also introduced a deprecation check step in CI. Before upgrading the compile SDK, a script would scan the codebase for any method that was deprecated in the new SDK and fail the build if found. That step alone caught two other deprecations that had been lurking in the codebase: one involving Notification.Builder.setPriority (deprecated in SDK 28, still usable but discouraged) and another with View.setBackgroundDrawable (deprecated in SDK 16, but the team had been using it for legacy compatibility). Both were easily fixed once identified.

“The rewrite forced us to understand our own tests,” the tech lead said. “We found tests that had been passing for years but were actually testing nothing—they threw exceptions that were swallowed by the harness.” One such test was checking a null return value from a method that always returned a non-null value, so the assertion was vacuously true. Another test had a typo in the expected value that never matched, but the harness caught the exception and reported it as a pass. These issues were invisible because the harness had no validation of test outcomes beyond pass/fail counts.

Lessons for the Wider Android Ecosystem

Google's API compatibility docs lag behind code. The deprecation of ActivityManager.getRunningTasks was noted in a blog post in 2019, but the official reference page did not show the removal SDK until months after SDK 34 shipped. Teams that rely on the reference documentation alone may miss removals. A better approach is to subscribe to the Android API diff reports, which are published for each SDK release and list all removed and deprecated APIs. However, these reports are not widely advertised; the team discovered them only after the incident.

Third-party CI providers rarely surface SDK deprecations. Most CI services run the Android SDK tools without additional linting. The team's CI provider did not fail the build when a deprecated method was used, even though the Android lint tool would have flagged it. Teams must configure lint to run as a build step and treat warnings as errors. This is a simple configuration change—adding lintOptions { abortOnError true } to the Gradle file—but many teams skip it because it can introduce false positives. The trade-off is between a slightly noisier build process and the risk of silent breakage. In this case, the team opted for the former and found that the additional warnings were manageable.

Open-source test frameworks rarely backport fixes for deprecated APIs. The team's mocking framework was abandoned; its last release predated SDK 30. The team had to fork it and apply patches themselves. That is a common pattern in the Android ecosystem, where many libraries are maintained by small teams or individuals. For example, the popular Robolectric framework has had several instances where deprecated Android APIs required community patches before official releases. Teams should evaluate the maintenance status of their dependencies and consider alternatives if a library appears dormant.

The recommended practice is an annual SDK audit with a deprecation test. Teams should upgrade their compile SDK at least once a year and run a full build with lint warnings treated as errors. That would have caught this breakage three years earlier, when the API was still available and the migration would have been less disruptive. Additionally, teams should maintain a changelog of deprecated APIs used in their codebase and review it before each SDK upgrade. This is a low-effort practice that can prevent major disruptions.

The team's story is not unique. A similar incident was documented for a TypeScript project where a custom transform broke after a language update. The pattern repeats across ecosystems: a deprecated API, a silent build, and a costly rewrite. Another team saved its monthly cloud budget with a five-euro plugin, showing that small investments in CI hygiene can prevent larger disasters. These stories underscore the importance of treating CI infrastructure as a first-class component of the software development lifecycle, not as an afterthought.

In the end, the team's test harness is now more maintainable, but the cost was high. The rewrite consumed four months of engineering time, roughly 1,200 person-hours, and added ongoing device costs. The team is now more cautious about SDK upgrades, but they acknowledge that the next breakage could come from a different direction—perhaps a Gradle plugin that stops being maintained, or a Firebase Test Lab API change. The platform giveth, and the platform taketh away.

Counter-Arguments: Was the Rewrite Necessary?

Not everyone on the team agreed that a full rewrite was the best path. One senior engineer argued for a more targeted fix: patch the harness to use ActivityManager.getRunningAppProcesses and update the mocking framework to use reflection to access the hidden classes. This approach would have taken roughly two weeks and avoided the migration to Bazel. However, the tech lead countered that reflection-based workarounds are fragile and can break with any Android runtime update. Indeed, Google has restricted reflection on hidden APIs since Android 9 (API 28), and future versions may block it entirely. The team voted to proceed with the rewrite, but the debate highlights a legitimate tension between short-term pragmatism and long-term health.

Another alternative was to stay on SDK 33 indefinitely and avoid the upgrade. This is a common strategy in enterprises with large codebases, but it carries its own risks: security patches and new features require the latest SDK, and app store policies may eventually mandate a higher target SDK. Google Play's target API level requirements have been increasing steadily, and by 2025, apps targeting below SDK 31 may be rejected. The team would have faced a forced upgrade eventually, likely under more pressure.

The rewrite also introduced new risks. Bazel, while powerful, has a steeper learning curve than Gradle, and the team spent several weeks just understanding its build rules and caching mechanisms. There were also integration issues with Firebase Test Lab: the Bazel test runner did not support all of Firebase's features out of the box, requiring custom wrappers. These costs are often underestimated in migration decisions.

Ultimately, the team's choice was context-dependent. For a team with a small codebase and a single deprecated API, a targeted fix might have sufficed. But for a team with a fragile harness and a high bus factor, the rewrite was a strategic investment. As the tech lead put it, “We paid the cost now so we don't have to pay it again next year.”

What Other Teams Can Do Differently

Based on this experience, the team recommends several concrete actions for Android CI pipelines:

• Run lint with abortOnError in CI. This catches deprecated API usage before it becomes a blocker. The team now uses a Gradle task that runs lint and fails the build if any warning is of type “Deprecation.”
• Maintain a deprecation log. Before each SDK upgrade, scan the codebase for all deprecated methods and decide whether to replace them. This can be automated with a simple script that parses lint output.
• Reduce bus factor. Ensure that at least two engineers understand the CI pipeline. The team now holds monthly “CI office hours” where engineers rotate responsibility for maintaining the harness.
• Test the harness itself. The team added unit tests for the CI scripts and a smoke test that runs a minimal build on a fresh environment to verify that the pipeline is functional.
• Consider managed test infrastructure. Firebase Test Lab or similar services reduce the overhead of maintaining emulator farms and provide more reliable results, though at a cost.

These steps are not silver bullets, but they would have prevented the team's specific breakage. The broader lesson is that CI infrastructure requires the same rigor as production code—including version control, testing, and documentation. Neglecting it is a form of technical debt that compounds over time.