Lock scratch directory during tool execution

[neonichu]

Currently, concurrent execution of SwiftPM can easily break the state in .build irreparably. This is especially in conjunction with tools using file watching to trigger package resolution which is something that e.g. the Swift VS Code plugin does.

We can resolve this by locking the .build directory for the duration of execution of any SwiftTool such that any following access will do a blocking wait until unlock. This is using the same infrastructure that we are already using for the shared repository cache. Unlike the shared cache, tool execution can take a long time, so we'll inform the user about the waiting process so that they can decide to cancel instead.

rdar://113964179

[neonichu]

I wonder if we need/want to do anything special for non-interactive execution (e.g. always bail if .build is locked) and also potentially a flag like swift build --no-wait-for-scratch-directory-lock.

[neonichu]

swiftlang/swift-tools-support-core#458
@swift-ci please test

[neonichu]

Not sure I understand the Linux failure:

[980/1149] Compiling Basics AbsolutePath.swift
/home/build-user/swiftpm/Sources/Basics/FileSystem/FileSystem+Extensions.swift:200:36: error: cannot convert value of type 'TSCAbsolutePath' (aka 'AbsolutePath') to expected argument type 'AbsolutePath'
        try self.withLock(on: path.underlying, type: type, blocking: blocking, body)
                                   ^
[981/1149] Compiling Basics FileSystem+Extensions.swift
/home/build-user/swiftpm/Sources/Basics/FileSystem/FileSystem+Extensions.swift:200:36: error: cannot convert value of type 'TSCAbsolutePath' (aka 'AbsolutePath') to expected argument type 'AbsolutePath'
        try self.withLock(on: path.underlying, type: type, blocking: blocking, body)
                                   ^

[neonichu]

Not sure I understand the Linux failure

Oh, of course this is because self-hosted jobs don't support cross repo testing.

[neonichu]

swiftlang/swift-tools-support-core#458
@swift-ci please test windows

[neonichu]

Windows failure looks related:

C:\Users\swift-ci\jenkins\workspace\swiftpm-PR-windows\swift-tools-support-core\Sources\TSCBasic\Lock.swift:117:37: error: cannot convert value of type 'Int32' to expected argument type 'Int'
        case .exclusive: dwFlags |= LOCKFILE_EXCLUSIVE_LOCK

[neonichu]

swiftlang/swift-tools-support-core#458
@swift-ci please test windows

[krzyzanowskim]

I wonder if we need/want to do anything special for non-interactive execution

nice! I proposed to lock a few years ago #2036 (comment), which, unfortunately, was rejected as not needed. I'm happy to see the idea finally made it through.

@neonichu The work is here in case that may be helpful in terms of scope #2037. One thing is that I believe Workspace should do the locking too, like when invoked by the sourcekit-lsp or other such.

[neonichu]

I wonder if we need/want to do anything special for non-interactive execution (e.g. always bail if .build is locked) and also potentially a flag like swift build --no-wait-for-scratch-directory-lock.

To answer myself here somewhat delayed, it doesn't seem as if we can do anything for non-interactive execution without requiring changes to the VS Code plugin (and potentially many other integrations that are shelling out to SwiftPM). By definition these integrations are non-interactive and if we don't make them wait they would fail in potentially many scenarios that work fine today.

[krzyzanowskim]

it doesn't seem as if we can do anything for non-interactive execution without requiring changes to the VS Code plugin

what exactly? shouldn't sourcekit-lsp at least be aware and avoid command line execution overlap?

[neonichu]

The VS Code plugin shells out directly to swift package resolve etc, outside of its use of sourcekit-lsp.

[krzyzanowskim]

that's exactly my point. these two may (and do) overlap without any lock