Fix high memory usage on apple devices

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 1.10.0
+
+- Add internal fork of Ktor Darwin HTTP engine with improved backpressure handling for large response bodies on Apple platforms.
+  This fixes out-of-memory crashes when syncing large payloads (hundreds of MB) on iOS/macOS.
+- Replaces unbounded channel with bounded channel (capacity 64) to prevent unbounded memory growth
+- Applies backpressure that propagates to the network layer, throttling data delivery based on processing speed
+- No API changes - this is a transparent improvement to the underlying HTTP handling
+
 ## 1.9.0
 
 - Updated user agent string formats to allow viewing version distributions in the new PowerSync dashboard.

core/build.gradle.kts

@@ -65,7 +65,8 @@ kotlin {
         }
 
         appleMain.dependencies {
-            implementation(libs.ktor.client.darwin)
+            // Use the local ktor-client-darwin module instead of the maven dependency
+            api(projects.internal.ktorClientDarwin)
 
             // We're not using the bundled SQLite library for Apple platforms. Instead, we depend on
             // static-sqlite-driver to link SQLite and have our own bindings implementing the

gradle.properties

@@ -19,7 +19,7 @@ development=true
 RELEASE_SIGNING_ENABLED=true
 # Library config
 GROUP=com.powersync
-LIBRARY_VERSION=1.9.0
+LIBRARY_VERSION=1.10.0
 GITHUB_REPO=https://github.com/powersync-ja/powersync-kotlin.git
 # POM
 POM_URL=https://github.com/powersync-ja/powersync-kotlin/

gradle/libs.versions.toml

@@ -91,6 +91,7 @@ ktor-client-darwin = { module = "io.ktor:ktor-client-darwin", version.ref = "kto
 ktor-client-okhttp = { module = "io.ktor:ktor-client-okhttp", version.ref = "ktor" }
 ktor-client-contentnegotiation = { module = "io.ktor:ktor-client-content-negotiation", version.ref = "ktor" }
 ktor-client-mock = { module = "io.ktor:ktor-client-mock", version.ref = "ktor" }
+ktor-network-tls = { module = "io.ktor:ktor-network-tls", version.ref = "ktor" }
 ktor-serialization-json = { module = "io.ktor:ktor-serialization-kotlinx-json", version.ref = "ktor" }
 kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serialization-json", version.ref = "serialization" }
 kotlinx-coroutines-core = { module = "org.jetbrains.kotlinx:kotlinx-coroutines-core", version.ref = "coroutines" }

internal/ktor-client-darwin/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2000-2023 JetBrains s.r.o.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

internal/ktor-client-darwin/README.md

@@ -0,0 +1,75 @@
+# Ktor Darwin Client (Internal Fork)
+
+This is an internal fork of the [Ktor Darwin HTTP engine](https://github.com/ktorio/ktor) with modifications to address memory issues when processing large HTTP response bodies on Apple platforms.
+
+## Why This Fork Exists
+
+### The Problem: Out of Memory (OOM) on Large Sync Payloads
+
+The upstream Ktor Darwin engine uses an unbounded channel to buffer incoming response data chunks from `NSURLSession`. When processing large sync payloads (hundreds of MBs), this causes:
+
+1. **NSURLSession delivers data faster than it can be processed** - chunks accumulate in the unbounded channel
+2. **Memory usage spikes dramatically** - we observed multi-GB allocations during sync operations
+3. **OOM crashes on iOS/macOS** - devices run out of memory before the response is fully processed
+
+This issue is specific to the Darwin engine because:
+
+- `NSURLSession` delivers data via delegate callbacks that cannot be paused
+- The upstream implementation uses `Channel.UNLIMITED` - buffering all chunks without backpressure
+- Other Ktor engines have natural backpressure mechanisms:
+  - **OkHttp**: Uses `BufferedSource.read()` which blocks until data is consumed
+  - **Apache/Apache5**: Uses `CapacityChannel` for explicit backpressure signaling
+
+### The Solution: Bounded Channel with Backpressure
+
+Our fork modifies `DarwinTaskHandler` to apply backpressure:
+
+```kotlin
+// Bounded channel instead of unbounded
+private val bodyChunks = Channel<NSData>(capacity = 64)
+
+fun receiveData(dataTask: NSURLSessionDataTask, data: NSData) {
+    val result = bodyChunks.trySend(data)
+    when {
+        result.isClosed -> dataTask.cancel()
+        result.isFailure -> {
+            // Buffer full - block to apply backpressure
+            runBlocking { bodyChunks.send(data) }
+        }
+    }
+}
+```
+
+Key changes:
+
+- **Limited channel capacity (64)** - prevents unbounded memory growth
+- **`runBlocking` on buffer full** - blocks the NSURLSession delegate thread, naturally slowing data delivery
+- **Backpressure propagates to NSURLSession** - the network layer throttles based on processing speed
+
+### Alternative Approaches Considered
+
+**Task Pause/Resume**: We considered using `NSURLSessionTask.suspend()` and `resume()` to pause data delivery when the buffer was full. However, this approach was rejected due to:
+
+- **Complexity** - managing pause/resume state across async boundaries added significant complexity
+- **Concurrency issues** - race conditions between pause signals and data delivery callbacks
+- **Data delivery timing** - the asynchronous nature of NSURLSession means data can still be delivered after calling `suspend()` on the task, which would require periodic draining and complex state management.
+- **Error-prone implementation** - the combination of these factors made the approach fragile and difficult to test
+
+The simpler bounded channel with `runBlocking` approach was chosen as it provides effective backpressure with minimal complexity and maintenance burden.
+
+## When to Update This Fork
+
+This fork should be updated if:
+
+- Ktor releases a fix for this issue upstream (track [ktor issues](https://github.com/ktorio/ktor/issues))
+- Security vulnerabilities are found in the Ktor Darwin engine
+- New Darwin-specific features are needed
+
+## Files Modified
+
+- `darwin/src/io/ktor/client/engine/darwin/internal/DarwinTaskHandler.kt` - Bounded channel + backpressure logic
+
+## References
+
+- [Original Ktor Darwin Engine](https://github.com/ktorio/ktor/tree/main/ktor-client/ktor-client-darwin)
+- PowerSync Kotlin SDK issue: OOM during large sync operations on iOS/macOS