Going Native - Swift
"Your second-hand bookseller is second to none in the worth of the treasures he dispenses."
Leigh Hunt
Coming home to roost
This is the completion of a series on calling native code from high-level languages. Here is a description of the native library I'm calling in this series.
Apple has used several languages for its operating system and devices, most notably Objective-C and Swift. But I read a few years ago that Swift had found some adoption in data analysis and Big Data applications because of its expressiveness and streaming features. Swift has been released in open source, so there are implementations for Linux and Windows in addition to MacOS. I did an Advent of Code in Swift one year, and enjoyed it. To wrap up this project of calling native code from high-level languages I decided to give Swift a try.
Getting Started
The interface for calling native code from Swift has changed recently. The mechanism is the Swift Package Manager, but the changes have meant some older references are out of date. One example that gave me hope, even though it didn't work was this blog post: Wrapping C Libraries in Swift.
The example that got me going was directly from the Swift Documentation on the Swift Package Manager, particularly using system libraries to call native code.
As an Apple-original language, I wasn't sure how it would translate to Windows. I was fairly confident in its applicability to Linux, though, so that's where I started. That meant writing a command line application, instead of an app: those are Mac-only.
1$ mkdir SwiftRMatrix
2$ cd SwiftRMatrix
3$ swift package init --type executable
4$ tree .
5.
6├── Package.swift
7└── Sources
8 └── SwiftRMatrix
9 └── SwiftRMatrix.swift
10
112 directories, 2 files
These commands set up a group of files and directories, the most important of which are Package.swift and Sources/SwiftRMatrix/SwiftRMatrix.swift. The latter is the entrypoint to the application, and the former is the directions for how to build the project. This is all that is needed to run "Hello, world!": you can do swift run at this point and see the message printed to the console.
Linking to native code is a matter of writing new modules and setting up dependencies among the modules in the project.
1$ mkdir Sources/CRashunal
2$ touch Sources/CRashunal/rashunal.h
3$ touch Sources/CRashunal/module.modulemap
rashunal.h:
1#import <rashunal.h>
module.modulemap:
1module CRashunal [system] {
2 umbrella header "rashunal.h"
3 link "rashunal"
4}
rashunal.h, which is distinct from the rashunal.h I wrote for the Rashunal project, is simply a transitive import to the native code, bringing all the declarations in the original rashunal.h into the Swift project. module.modulemap emphasizes this by saying that rashunal.h is an umbrella header, and that the code will link the rashunal library. At this point, CRashunal (the Swift project) can be imported into Swift code and used.
Package.swift:
1// swift-tools-version: 6.2
2// The swift-tools-version declares the minimum version of Swift required to build this package.
3
4import PackageDescription
5
6let package = Package(
7 name: "SwiftRMatrix",
8 dependencies: [],
9 targets: [
10 // Targets are the basic building blocks of a package, defining a module or a test suite.
11 // Targets can depend on other targets in this package and products from dependencies.
12 .systemLibrary(
13 name: "CRashunal"
14 ),
15 .executableTarget(
16 name: "SwiftRMatrix",
17 dependencies: ["CRashunal"],
18 path: "Sources/SwiftRMatrix"
19 ),
20 ]
21)
SwiftRMatrix.swift:
1// The Swift Programming Language
2// https://docs.swift.org/swift-book
3import Foundation
4
5@main
6struct SwiftRMatrix {
7 public func run() throws {
8 let r: UnsafeMutablePointer<CRashunal.Rashunal> = n_Rashunal(numericCast(1), numericCast(2))
9 print("{\(r.pointee.numerator),\(r.pointee.denominator)}")
10 }
11}
I like that Swift distinguishes between mutable and immutable pointers (UnsafeMutablePointer and UnsafePointer), and uses generics to indicate what the pointer is to. Swift also has an OpaquePointer when the fields of a struct are not imported, like an RMatrix. I'll come back to that later. The pointee field to access the fields of the struct is an additional bonus.
ChatGPT pointed me to memory safety early on, so I learned quickly how to access the standard library on the different platforms. Swift recognizes C-like compiler directives, so accessing it was a simple matter of importing the right native libraries. For Windows, it's a part of the platform, so no special import is needed.
1#if os(Linux)
2import Glibc
3#elseif os(Windows)
4
5#elseif os(macOS)
6import Darwin
7#else
8#error("Unsupported platform")
9#endif
10...
11let r: UnsafeMutablePointer<CRashunal.Rashunal> = n_Rashuna(numericCast(1), numericCast(2))
12print("{\(r.pointee.numerator),\(r.pointee.denominator)}")
13free(r)
And that's it, for code. The devil, of course, is in the compiling and linking.
A chain is only as strong as its weakest link
Swift Package Manager uses several sources to find libraries, but none of them seemed to match my particular use case. The closest was to make use of pkg-config. The more I read about it, the more it seemed to be an industry standard, and that Rashunal and RMatrix would benefit by taking advantage of it. So I broke my rule that I established earlier and decided to enhance the libraries.
Fortunately, it wasn't too painful. Telling Rashunal to write to pkg-config was only a few lines added to rashunal/CMakeLists.txt:
1+set(PACKAGE_NAME rashunal)
2+set(PACKAGE_VERSION 0.0.1)
3+set(PACKAGE_DESC "Rational arithmetic library")
4+set(PKGCONFIG_INSTALL_DIR "${CMAKE_INSTALL_LIBDIR}/pkgconfig")
5+
6+configure_file(
7+ ${CMAKE_CURRENT_SOURCE_DIR}/rashunal.pc.in
8+ ${CMAKE_CURRENT_BINARY_DIR}/${PACKAGE_NAME}.pc
9+ @ONLY
10+)
11+
12 add_library(rashunal SHARED src/rashunal.c src/rashunal_util.c)
13...
14+install(
15+ FILES ${CMAKE_CURRENT_BINARY_DIR}/rashunalConfig.cmake
16+ DESTINATION lib/cmake/rashunal
17+)
18+
19+install(
20+ FILES ${CMAKE_CURRENT_BINARY_DIR}/${PACKAGE_NAME}.pc
21+ DESTINATION ${PKGCONFIG_INSTALL_DIR}
22 )
The first block is toward the top of CMakeLists.txt, and the second is toward the bottom.
The configure_file directive needs a template for the pc file that will be written. The template has placeholders set of by '@' that will be filled in during the build process.
rashunal.pc.in:
1prefix=@CMAKE_INSTALL_PREFIX@
2exec_prefix=${prefix}
3libdir=${exec_prefix}/@CMAKE_INSTALL_LIBDIR@
4includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@
5
6Name: @PACKAGE_NAME@
7Description: @PACKAGE_DESC@
8Version: @PACKAGE_VERSION@
9Libs: -L${libdir} -l@PACKAGE_NAME@
10Cflags: -I${includedir}
During installation the newly-written rashunal.pc file will be written to a platform-standard location on disk.
After making those changes, building, compiling, and installing, pkg-config was able to tell me something about the Rashunal library:
1$ rm -rf build
2$ mkdir build
3$ cd build
4$ cmake ..
5$ make && sudo cmake --install .
6$ ls /usr/local/lib/pkgconfig
7rashunal.pc
8$ cat /usr/local/lib/pkgconfig/rashunal.pc
9prefix=/usr/local
10exec_prefix=${prefix}
11libdir=${exec_prefix}/lib
12includedir=${prefix}/include
13
14Name: rashunal
15Description: Rational arithmetic library
16Version: 0.0.1
17Libs: -L${libdir} -lrashunal
18Cflags: -I${includedir}
19$ pkg-config --cflags rashunal
20-I/usr/local/include
21$ pkg-config --libs rashunal
22-L/usr/local/lib -lrashunal
Notice the new command to install the project: apparently this is the more modern and more approved way to do it nowadays. The bash output means that the declarations of the Rashunal library can be found at /usr/local/include and the binaries at /usr/local lib.
Now the Swift Package Manager can be told just to consult pkg-config for the header and binary location of any system libraries it's attempting to build. It's not necessary, but the examples I saw recommended adding some suggestions for how to install Rashunal if it's not present. I haven't looked into what it takes to package a library for apt or brew, but I'm pretty sure this is how they are consumed:
Package.swift:
1.systemLibrary(
2 name: "CRashunal",
3 pkgConfig: "rashunal",
4 providers: [
5 .apt(["rashunal"]),
6 .brew(["rashunal"]),
7 ],
8)
Then the Swift project could be built and run:
1$ swift build
2$ swift run SwiftRMatrix
3{1,2}
And rinse and repeat for RMatrix. There is nothing new in building the RMatrix pkg-config files or linking to it from Swift, except for the dependency on Rashunal in the template for RMatrix:
rmatrix.pc.in
1prefix=@CMAKE_INSTALL_PREFIX@
2exec_prefix=${prefix}
3libdir=${exec_prefix}/@CMAKE_INSTALL_LIBDIR@
4includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@
5
6Name: @PACKAGE_NAME@
7Description: @PACKAGE_DESC@
8Version: @PACKAGE_VERSION@
9Requires: rashunal
10Libs: -L${libdir} -l@PACKAGE_NAME@
11Cflags: -I${includedir}
I started to look into removing that hardcoded dependency and getting it from the link libraries in CMakeLists.txt, but that quickly started to grow big and nasty, so I abandoned it. ChatGPT assured me that was common, especially for small projects.
Crossing the operating system ocean
Trying to do this on MacOS, I ran into my old nemesis SIP. Fortunately, the solution here was similar to the solution I followed there. The Swift command at /usr/bin/swift was protected by SIP, but the executable generated by the swift build command wasn't:
1% swift build -Xlinker -rpath -Xlinker /usr/local/lib
2% swift run .build/debug/SwiftRMatrix
3{1,2}
What is astonishing is that, with one more testy exchange with ChatGPT, I also got it to work on Windows. I still don't understand what was the difference with Linux and MacOS or how this changed things on Windows, but I had to make an additional change to Rashunal's CMakeLists.txt and the cmake command to build RMatrix:
rashunal/CMakeLists.txt
1if (WIN32)
2 set_target_properties(rashunal PROPERTIES
3 ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
4 RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
5 )
6endif()
1>cmake .. -G "NMake Makefiles" ^
2More? -DCMAKE_BUILD_TYPE=Release ^
3More? -DCMAKE_INSTALL_PREFIX=C:/Users/john.todd/local/rmatrix ^
4More? -DCMAKE_PREFIX_PATH=C:/Users/john.todd/local/rashunal ^
5More? -DCMAKE_C_FLAGS_RELEASE="/MD /O2 /DNDEBUG"
6>nmake
7>nmake install
Then the Swift application could be built and run from the command line, albeit with a few additional linker switches. This also needs to be done from a Powershell or DOS window with Admin rights because, even though it only changes the local project directory, it seems to write to a protected directory.
1> swift build `
2>> -Xcc -IC:/Users/john.todd/local/rashunal/include `
3>> -Xcc -IC:/Users/john.todd/local/rmatrix/include `
4>> -Xlinker /LIBPATH:C:/Users/john.todd/local/rashunal/lib `
5>> -Xlinker /LIBPATH:C:/Users/john.todd/local/rmatrix/lib `
6>> -Xlinker /DEFAULTLIB:rashunal.lib `
7>> -Xlinker /DEFAULTLIB:rmatrix.lib `
8>> -Xlinker /DEFAULTLIB:ucrt.lib
9> ./.build/debug/SwiftRMatrix.exe
10{1,2}
Cleaning up the guano
My last task was to abstract the native calls away from the main application. To do this I wrote a Models module that wrapped the native Rashunal, RMatrix, and Gauss Factorization structs.
Sources/Model/Model.swift
1public class Rashunal: CustomStringConvertible {
2 var _rashunal: UnsafePointer<CRashunal.Rashunal>
3
4 public init(_ numerator: Int, _ denominator: Int = 1) {
5 _rashunal = UnsafePointer(n_Rashunal(numericCast(numerator), numericCast(denominator)))
6 }
7
8 public init(_ data: [Int]) {
9 _rashunal = UnsafePointer(n_Rashunal(numericCast(data[0]), data.count > 1 ? numericCast(data[1]) : 1))
10 }
11
12 public var numerator: Int { Int(_rashunal.pointee.numerator) }
13
14 public var denominator: Int { Int(_rashunal.pointee.denominator) }
15
16 public var description: String {
17 return "{\(numerator),\(denominator)}"
18 }
19
20 deinit {
21 free(UnsafeMutablePointer(mutating: _rashunal))
22 }
23}
What gets returned from the native n_Rashunal call is a Swift UnsafeMutablePointer<CRashunal.Rashunal>. I wanted them to be immutable wherever possible, so I cast it to an UnsafePointer<CRashunal.Rashunal> in both the constructors. Swift makes property definition and string representations easy and natural. The deinit method calls the native standard library's free method to release the native memory allocated by Rashunal. This makes cleanup and memory hygiene easy.
Sources/Model/Model.swift
1public class RMatrix: CustomStringConvertible {
2 var _rmatrix: OpaquePointer
3
4 private init(_ rmatrix: OpaquePointer) {
5 _rmatrix = rmatrix
6 }
7
8 public init(_ data: [[[Int]]]) {
9 let height = data.count
10 let width = data.first!.count
11
12 let rashunals = data.flatMap {
13 row in row.map {
14 cell in n_Rashunal(numericCast(cell[0]), cell.count > 1 ? numericCast(cell[1]) : 1)
15 }
16 }
17 let ptrArray = UnsafeMutablePointer<UnsafeMutablePointer<CRashunal.Rashunal>?>.allocate(capacity: rashunals.count)
18 for i in 0..<rashunals.count {
19 ptrArray[i] = rashunals[i]
20 }
21 defer { r in
22 rashunals.forEach { free(r) }
23 ptrArray.deallocate()
24 }
25
26 _rmatrix = new_RMatrix(numericCast(height), numericCast(width), ptrArray)
27 }
28
29 public var height: Int { Int(RMatrix_height(_rmatrix)) }
30
31 public var width: Int { Int(RMatrix_width(_rmatrix)) }
32
33...
34 public var description: String {
35 (1...height).map { i in
36 "[ " + (1...width).map { j in
37 let cellPtr: UnsafePointer<CRashunal.Rashunal> = RMatrix_get(_rmatrix, i, j)
38 let rep = "{\(cellPtr.pointee.numerator),\(cellPtr.pointee.denominator)}"
39 free(UnsafeMutablePointer<CRashunal.Rashunal>(mutating: cellPtr))
40 return rep
41 }.joined(separator: " ") + " ]"
42 }.joined(separator: "\n")
43 }
44
45 deinit {
46 free_RMatrix(_rmatrix)
47 }
48}
Unsurprisingly, RMatrix was the hardest of these to get right. The private constructor is used in the factor method as a convenience method to initialize a Swift RMatrix. The other constructor is used to initialize a matrix from the familiar 3D array of Ints. I get the height and width from the first two dimensions of the input array, then use the n_Rashunal method to construct a list of native Rashunal structs as UnsafeMutablePointer<CRashunal.Rashunal>s. As before, new_RMatrix expects an array of pointers to structs, but the rashunals array is in managed memory, not native memory. So I allocate and fill an array of pointers to the Rashunal structs in native memory. ChatGPT suggested I add the defer block in case new_RMatrix abends for any reason. Because the RMatrix struct is declared but not defined in rmatrix.h, what is automatically returned is an OpaquePointer, which is just fine with me.
Properties defer to the encapsulated _rmatrix pointer, and the string description method makes full use of Swift's stream processing capabilities. deinit calls the RMatrix library's free_RMatrix method.
After all that, factoring a matrix and the GaussFactorization struct are pretty routine.
Sources/Model/Model.swift
1public struct GaussFactorization {
2 public var PInverse: RMatrix
3 public var Lower: RMatrix
4 public var Diagonal: RMatrix
5 public var Upper: RMatrix
6
7 public init(PInverse: RMatrix, Lower: RMatrix, Diagonal: RMatrix, Upper: RMatrix) {
8 self.PInverse = PInverse
9 self.Lower = Lower
10 self.Diagonal = Diagonal
11 self.Upper = Upper
12 }
13}
14
15public class RMatrix: CustomStringConvertible {
16...
17 public func factor() -> GaussFactorization {
18 let gf = RMatrix_gelim(_rmatrix)!
19 let sgf = GaussFactorization(
20 PInverse: RMatrix(gf.pointee.pi),
21 Lower: RMatrix(gf.pointee.l),
22 Diagonal: RMatrix(gf.pointee.d),
23 Upper: RMatrix(gf.pointee.u)
24 )
25 free(gf)
26 return sgf
27 }
28}
Calling the native method RMatrix_gelim returns a newly-allocated struct pointing to four newly-allocated matrices. The matrices are passed to the RMatrix constructor, so that the class takes responsibility for managing their memory. The native struct itself is freed by the RMatrix factor method before returning the Swift struct.
The driver class has no import to native code, and all the allocations look just like Swift objects.
1import ArgumentParser
2import Foundation
3import Model
4
5enum SwiftRMatrixError: Error {
6 case runtimeError(String)
7}
8
9@main
10struct SwiftRMatrix: ParsableCommand {
11 @Option(help: "Specify the input file")
12 public var inputFile: String
13
14 public func run() throws {
15 let url = URL(fileURLWithPath: inputFile)
16 var inputText = ""
17 do {
18 inputText = try String(contentsOf: url, encoding: .utf8)
19 } catch {
20 throw SwiftRMatrixError.runtimeError("Error reading file [\(inputFile)]")
21 }
22 let data = inputText
23 .split(whereSeparator: \.isNewline)
24 .map { $0.trimmingCharacters(in: .whitespaces) }
25 .map { line in line.split(whereSeparator: { $0.isWhitespace })
26 .map { token in token.split(separator: "/").map { Int($0)! } }
27 }
28 let m = Model.RMatrix(data)
29 print("Input matrix:")
30 print(m)
31
32 let factor = m.factor()
33 print("Factors into:")
34 print("PInverse:")
35 print(factor.PInverse)
36
37 print("Lower:")
38 print(factor.Lower)
39
40 print("Diagonal:")
41 print(factor.Diagonal)
42
43 print("Upper:")
44 print(factor.Upper)
45 }
46}
1$ swift run SwiftRMatrix --input-file /home/john/workspace/rmatrix/driver/example.txt
2[1/1] Planning build
3Building for debugging...
4[11/11] Linking SwiftRMatrix
5Build of product 'SwiftRMatrix' complete! (1.17s)
6Input matrix:
7[ {-2,1} {1,3} {-3,4} ]
8[ {6,1} {-1,1} {8,1} ]
9[ {8,1} {3,2} {-7,1} ]
10Factors into:
11PInverse:
12[ {1,1} {0,1} {0,1} ]
13[ {0,1} {0,1} {1,1} ]
14[ {0,1} {1,1} {0,1} ]
15Lower:
16[ {1,1} {0,1} {0,1} ]
17[ {-3,1} {1,1} {0,1} ]
18[ {-4,1} {0,1} {1,1} ]
19Diagonal:
20[ {-2,1} {0,1} {0,1} ]
21[ {0,1} {17,6} {0,1} ]
22[ {0,1} {0,1} {23,4} ]
23Upper:
24[ {1,1} {-1,6} {3,8} ]
25[ {0,1} {1,1} {-60,17} ]
26[ {0,1} {0,1} {1,1} ]
Reflection
Wow, that turned out a lot better than I expected. I thought this would be possible on Linux and MacOS. To be able to get it to work on Windows too was a pleasant surprise. I really like the Swift language: it is expressive and concise and makes really good use of streaming approaches. I hope I get to use it to make money sometime.