Migration Plan: swift-corelibs-foundation to swift-foundation

Related Projects Foundation
1

Migration Plan: swift-corelibs-foundation to swift-foundation

As we progress with our plans for the future of Foundation, it is time to consider in more detail how we will migrate the Swift ecosystem from swift-corelibs-foundation, which is part of the Swift toolchain, to swift-foundation, which is a standalone package. Please let us know your thoughts and questions in this thread.

First, let's define the requirements for a transition:

  • Packages which use swift-corelibs-foundation today must continue to build and run.

  • Packages must be able to adopt the new swift-foundation package -- even if they depend on packages which use swift-corelibs-foundation, or packages that use swift-corelibs-foundation depend on them.

After some experimentation, we believe we have a path forward. At a high level, the plan is:

  1. Enable building swift-corelibs-foundation as a package, in addition to building as part of the toolchain.

  2. Make the swift-corelibs-foundation package depend on the swift-foundation package. Re-export the FoundationEssentials and FoundationInternationalization modules from swift-corelibs-foundation.

  3. Delete now-duplicated types from swift-corelibs-foundation.

With these changes, everyone in the ecosystem, no matter which of the packages they import, will be using one implementation of core types like Date, Data, and JSONDecoder.

Detailed implementation

The main principle behind this plan is that there must be only one definition of the core types in a process. If there is only one Date, for example, then there can be no mismatch between types that are otherwise identical but coming from two different modules. We took the same approach for the Darwin Foundation.framework, which has been refactored to use the same Swift source code as swift-foundation and provides a single implementation for the entire OS.

Module availability

This new structure gives package authors the flexibility to choose which dependency they wish to use. For example, an author can choose to minimize code size on non-Darwin platforms if features like networking or internationalization are not required.

Module Package Note
FoundationEssentials swift-foundation also re-exported from swift-corelibs-foundation
FoundationInternationalization swift-foundation also re-exported from swift-corelibs-foundation
FoundationNetworking swift-corelibs-foundation
FoundationXML swift-corelibs-foundation
Foundation (Non-Darwin) swift-corelibs-foundation Does not include Networking or XML
Foundation (Umbrella, Darwin) Foundation.framework (SDK) Includes all Foundation features

Preliminary testing shows that a simple command-line utility built for Ubuntu 22.04 can reduce its binary size by approximately 60 MB when using FoundationEssentials vs FoundationInternationalization, assuming it does not need the internationalization functionality provided by the latter module. The size reduction comes primarily from omitting the locale databases included with the ICU library.

swift-corelibs-foundation will re-export FoundationEssentials and FoundationInternationalization from Foundation. This will provide all of the new API from those modules (notably FormatStyle) for clients of swift-corelibs-foundation without requiring them to transition immediately to the new modules. It is possible that differences between swift-corelibs-foundation and swift-foundation will cause some behavioral change at runtime, but we expect the risk to be only slightly more than the library fixing bugs or adding features with any other regular Swift update.

In short, swift-corelibs-foundation will play the role of a compatibility layer between the new implementations and existing packages. The focus of our development will be on the swift-foundation package.

XCTest

A branch of swift-corelibs-xctest will be added that depends on the swift-corelibs-foundation package instead of the toolchain swift-corelibs-foundation. Test targets will use this branch to ensure the prescence of only one of the core Foundation types at runtime.

Swift Package Manager changes

In order to maintain compatibility with existing packages, SwiftPM will be updated to automatically add an implict dependency on the swift-corelibs-foundation package when it detects the use of Foundation in a package. If an explicit dependency on swift-corelibs-foundation or swift-foundation is specified, that will take precedence. In a future swift-tools-version, SwiftPM will no longer add the dependency and it must be specified by the Package.swift file, the same as any other package dependency. A swift-corelibs-xctest package dependency must also be added for test targets.

We also anticipate some improvements to the PackageDescription API to allow more C flags to be passed through to the compiler without the use of unsafeFlags. For example, CoreFoundation requires the use of -fconstant-cfstrings.

Cross-platform compatibility

Many contributions have been made to swift-corelibs-foundation to port the project to non-Darwin, non-Linux platforms. As we migrate the project to rebase itself on top of swift-foundation, we will preserve these changes where appropriate to maintain swift-corelibs-foundation's cross-platform compatibility. We anticipate the need for help from the community to make sure we have testing and validation coverage for platforms that are not part of Swift's normal continuous integration or release testing process. In many cases, this will require code written in C for CoreFoundation to be rewritten in Swift for swift-foundation.

Using Foundation in Package.swift and plugins

In order to support import Foundation in Package.swift and plugins, a version of the Foundation library will be available in the toolchain for use at package manifest parsing time only. This version of Foundation is not intended to be used at runtime. Our goal is to build this library from the same sources as the package version of swift-corelibs-foundation, using cmake as it does today. To simplify maintenance of this code, we may choose to narrow the types available to the most common APIs used in Package.swift files.

45 Likes
2

This is great news! I'm looking forward to the transition as it would ease other issues that users of Swift run into on Windows (with not being able to get improvements to Foundation without a new toolchain release).

One thing that seems uncovered is the testing aspect. swift-corelibs-foundation is currently tested on Windows but swift-foundation does not include testing. This seems like it would be a strict regression in terms of coverage. Is that going to be addressed prior to the migration?

4 Likes
3

We do need to figure out how to get Windows testing up and running for swift-foundation. I don't think we can regress any of swift-corelibs-foundation's targets during this 'rebasing'.

I know we have work to do here on making sure that Windows-specific behavior migrates into the swift-foundation library from swift-corelibs-foundation where appropriate as well. One example off the top of my head is the time zone database discovery. The same is true for some other platforms like WASM.

2 Likes
4

This is great news!

:tada:

3 Likes
5

Great news. Thanks for your team and Swift community's hard work.

A little question is when can we (the downstream non-Darwin platform developer) start to migrate? Do we need to wait for Swift 6 or can I start it right now with Swift 5.10?

The background is I'm encountering some strange Linux+Foundation compiler crash.
And I'm struggling to fix it.

Wondering if I can just instead start to migrate to using the new Foundation right now.

1 Like
6

It’s possible (we’ve cut our own fork / release of the new foundations main a while ago), but requires a little bit of work IIRC (perhaps related to us using system foundation on Darwin and the new one on Linux only, as the system one on Darwin already had predicates). We wanted to be able to use the new predicates and wanted to be able to provide feedback early so… it all depends on risk/reward, but so far it’s worked quite nicely for us.

2 Likes
7

I'm really excited to see swift-foundation replace swift-corelibs-foundation! Thanks for all the work getting closer to this goal.

I decided to try importing swift-foundation into our monorepo to see how well it would work as a replacement in its current state to anywhere that we're using Foundation on Linux. Unfortunately, my efforts have hit a bit of a wall: we have a "one-version policy" for third-party libraries to prevent issues where a binary in a large build graph might inadvertently try to link in two versions of the same library. So, I was trying to link FoundationInternationalization against the stock version of ICU that we have, but the vendored copy in GitHub - apple/swift-foundation-icu contains a great number of Apple-specific changes that FoundationInternationalization depends on.

Are there any plans to get swift-foundation building with a stock ICU by either removing or upstreaming those changes?

6 Likes
8

Seems like a great step in the right direction! Would the long-term goal still be to deprecate swift-corelibs-foundation in favor of swift-foundation? I've found it difficult to explain all the different versions of Foundation that now exist, and I'd love if things became simpler rather than more complex.

2 Likes
9

It is still a long-term goal, but with this approach swift-corelibs-foundation will be providing a superset of swift-foundation API by definition. It will probably extend the lifetime of the corelibs version by making it less urgent to switch.

I think it is reasonable for us to provide some more guidance on which library to import as part of our various readme files and docs, though.

2 Likes
10

A great question. Ultimately comes down to a tradeoff -- do we want a higher fidelity implementation (behavior matching across all platforms) or do we want to use common platform libraries? Ultimately Foundation relies upon ICU to provide specifics about how values are formatted. And those specifics change from time to time, with changes in opinion about what the best way to display those values are. In macOS 14, for example, ICU changed to put a non-breaking space between the time and any AM/PM designation.

We also have extensions to ICU for additional features or performance. Just recently I was looking at adding one to ICU's TimeZone API to dramatically improve the efficiency of calculating time zone offsets. Bundle has an extension in ICU to provide a kind of localization-aware ordering of languages so that Bundle can provide better fallback paths when an app's available localizations and a user's preferences do not exactly match.

Some of these we hope to upstream, but ICU is of course its own project and these upstreams can take months or years -- even if they are desired by the upstream project in general.

With these considerations, we chose for swift-foundation to go in a different direction than swift-corelibs-foundation -- and provide the swift-foundation-icu library to ensure we have consistency first.

4 Likes
11

Thanks for explaining your reasoning!

If it's unlikely that swift-foundation-icu will be able to converge completely with mainline ICU, would you be open to renaming the symbols in the vendored copy of ICU so that they don't collide with the mainline library? I don't think this is necessarily a hypothetical limited to my company; if someone builds a statically linked Linux binary that depends on swift-foundation, which also includes an ICU dependency from some non-Swift source, they could run into the same issue. (In fact, we've already seen this problem with the use of LLVMSupport inside the Swift runtime, which has since been resolved by wrapping the symbols in a new inline namespace.)

2 Likes
12

Renaming the symbols sounds like a reasonable path. In fact we do want this ICU implementation to remain private to Foundation so we can keep them coupled.

Hopefully there is some way we could do that globally instead of having to change every function in the library. If you have any suggestions on how to do that, please let us know (or opening a pull request of course).

2 Likes
13

There are a lot of moving parts right now and we're working hard on trying to get it lined up so it is as easy as possible. I think we want to avoid prematurely tagging the repo and giving people a bad first impression, if it doesn't work well enough or is complete enough.

In fact, we do have a dependency on the Swift 6 compiler for the rebased swift-corelibs-foundation because of an old restriction (@Douglas_Gregor helpfully fixed it already here).

If you're curious about progress, here are some links:

The package branch of swift-corelibs-foundation
The package branch of swift-corelibs-xctest

We are working on getting CI up for these branches, which will be long-lived.

Changes for swift-foundation will go into its main branch.

1 Like
14

I think we should be able to take advantage of ICU's existing symbol renaming functionality here; luckily the library was designed with these kinds of concerns in mind. IIRC, the C++ symbols get put into different namespaces if configured with renaming, and the C functions get renamed and #defines are used to make them compatible with the original names.

The main issues that immediately jump out to me are:

  • We should make sure that any functions added by Apple are covered by urename.h so they end up with the same prefix (not strictly necessary since a new function would obviously not collide with a library that doesn't have it, but it's probably best to be consistent)
  • The #defines that rename the functions won't be imported into Swift, so the Swift code will have to be updated to use the alternative names instead of the original.

I don't think either of those issues are major problems, just grunt work. I'm happy to spend some time to see if I can get it working smoothly.

1 Like
15

Ok, thanks for looking into this. One additional thought - we do need to maintain source compatibility with the ICU on Darwin, so we may end up having to put some stub functions in. @icharleshu has mainly been maintaining the swift-foundation-icu repo, so he may have some other thoughts.

1 Like
16

Besides all these shiny new things, I'l like to stress that it's extremely important to make sure that a) software currently using swift-corelibs-foundation keeps running and b) software currently using swift-corelibs-foundation can use the new implementations, where they are available.

Since the new Foundation lacks and will probably never get (for me) critical infrastructure, I have to continue using swift-corelibs-foundation for a long time coming.

17

Good to hear this is moving forward. I think this will make providing Foundation for Android easier, as I can cross-compile the new swift-foundation-icu package using SwiftPM, rather than using whatever antiquated build scripts ICU has.

I just tried building the swift-foundation package for the first time using Swift 5.10 natively on Android and was surprised to see only three uses of #if os(Linux) in Sources/. Is this because much of the platform-specific code has not been ported yet? For comparison, I see hundreds of #if TARGET_OS_LINUX in CoreFoundation.

Also, are there any plans to add visionOS and bridgeOS to the list of defined OS's in the OSS toolchain? Otherwise, I see a lot of warnings like this with 5.10:

swift-foundation/Sources/FoundationEssentials/Data/Data+Error.swift:22:9: warning: unknown operating system for build configuration 'os'
#if !os(bridgeOS)
        ^
swift-foundation/Sources/FoundationEssentials/Data/Data+Error.swift:22:9: note: did you mean 'iOS'?
#if !os(bridgeOS)
        ^~~~~~~~
        iOS
18

With respect to Linux, the swift-foundation package builds and tests successfully for all of Swift's current Linux targets. I don't think comparing the number of ifdefs in the C files to what we have in the new Swift implementation is going to yield much useful information. For example, I see a lot of them in places like CFLocale.c and CFDate.c for some reason in CoreFoundation, but that C code is simply not present in swift-foundation, having been replaced by cross-platform Swift code.

Also, are there any plans to add visionOS and bridgeOS to the list of defined OS's in the OSS toolchain?

It looks like visionOS at least is resolved for Swift 6. I may just move the bridgeOS one out of this repository completely, but it would be nicer if the compiler suppressed the warning when it's inside another ifdef block that prevents it from compiling in the first place.

1 Like
19

I have not looked into how much of the old API surface the new package covers in an equivalent way, just wondered if you focused on the cross-platform APIs first.

No, I can reproduce with the latest trunk snapshot on linux, and the OSS compiler source shows no change to include these new OS's:

> ../swift-DEVELOPMENT-SNAPSHOT-2024-03-31-a-ubi9/usr/bin/swift build --build-tests
---snip---
/home/fina/swift-foundation/Sources/FoundationEssentials/Data/Data+Error.swift:22:9: warning: unknown operating system for build configuration 'os'
20 │ internal func logFileIOErrno(_ err: Int32, at place: String) {
21 │ #if FOUNDATION_FRAMEWORK
22 │ #if !os(bridgeOS)
   │         ├─ warning: unknown operating system for build configuration 'os'                                                                                        │         ╰─ note: did you mean 'iOS'?
23 │     let errnoDesc = String(cString: strerror(err))
24 │     Logger(_NSOSLog()).error("Encountered \(place) failure \(err) \(errnoDesc)")
---snip---
/home/fina/swift-foundation/Tests/FoundationEssentialsTests/ProcessInfoTests.swift:93:16: warning: unknown operating system for build configuration 'os'            91 │     func testOperatingSystemVersion() {
 92 │         let version = _ProcessInfo.processInfo.operatingSystemVersion
 93 │         #if os(visionOS)
    │                ├─ warning: unknown operating system for build configuration 'os'
    │                ├─ note: did you mean 'iOS'?
    │                ╰─ note: did you mean 'Windows'?
 94 │         let expectedMinMajorVersion = 1
 95 │         #else

The good news is that there are only three instances of calling these new OS's in swift-foundation:

> ag "(vision|bridge)OS" .
Tests/FoundationInternationalizationTests/CalendarRecurrenceRuleTests.swift
20:@available(macOS 15, iOS 18, tvOS 18, watchOS 11, visionOS 2, *)

Tests/FoundationEssentialsTests/ProcessInfoTests.swift
93:        #if os(visionOS)

Sources/FoundationEssentials/Data/Data+Error.swift
22:#if !os(bridgeOS)

The bad news is that SwiftPM will dump such warnings and errors out over and over again, likely once for each file in a target. I wish somebody on the SwiftPM team would work on reducing that.

1 Like
20

One item that I think that is important to bring up is the build - we will absolutely need to have a CMake based build system for this. There are currently dependencies on swift-corelibs-foundation in the toolchain, and @Douglas_Gregor has already shared the policy that the toolchain shall remain buildable with CMake - and is something that would prevent Windows from building. Has this already been taken into consideration?