Skip to content

Reduce ratchet memory usage for multi-module projects #1426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Dec 31, 2022
Merged

Reduce ratchet memory usage for multi-module projects #1426

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6972365
Reduce ratchet memory usage for multi-module projects
laurentgo Dec 15, 2022
f6f37f4
Update CHANGES.md
laurentgo Dec 15, 2022
38ca03b
spotlessApply
nedtwigg Dec 18, 2022
a4d5526
Simplify GitRatchet cache logic
laurentgo Dec 19, 2022
e46b9a1
Fix changelogs.
nedtwigg Dec 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,16 @@ This document is intended for Spotless developers.
We adhere to the [keepachangelog](https://keepachangelog.com/en/1.0.0/) format (starting after version `1.27.0`).

## [Unreleased]
### Fixed
* Improve memory usage when using git ratchet ([#1426](https://github.com/diffplug/spotless/pull/1426))

## [2.31.0] - 2022-11-24
### Added
* `importOrder` now support groups of imports without blank lines ([#1401](https://github.com/diffplug/spotless/pull/1401))
### Fixed
* Don't treat `@Value` as a type annotation [#1367](https://github.com/diffplug/spotless/pull/1367)
* Support `ktlint_disabled_rules` in `ktlint` 0.47.x [#1378](https://github.com/diffplug/spotless/pull/1378)
* Share git repositories across projects when using ratchet ([#1426](https://github.com/diffplug/spotless/pull/1426))
### Changes
* Bump default `ktfmt` version to latest `0.40` -> `0.41` ([#1340](https://github.com/diffplug/spotless/pull/1340))
* Bump default `scalafmt` version to latest `3.5.9` -> `3.6.1` ([#1373](https://github.com/diffplug/spotless/pull/1373))
Expand Down
2 changes: 1 addition & 1 deletion build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ spotless {
trimTrailingWhitespace()
endWithNewline()
}
}
}
50 changes: 9 additions & 41 deletions lib-extra/src/main/java/com/diffplug/spotless/extra/GitRatchet.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright 2020-2021 DiffPlug
* Copyright 2020-2022 DiffPlug
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand All @@ -19,7 +19,6 @@
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;

import javax.annotation.Nullable;
Expand Down Expand Up @@ -137,7 +136,7 @@ private static boolean worktreeIsCleanCheckout(TreeWalk treeWalk) {
private final static int INDEX = 1;
private final static int WORKDIR = 2;

Map<Project, Repository> gitRoots = new HashMap<>();
Map<File, Repository> gitRoots = new HashMap<>();
Table<Repository, String, ObjectId> rootTreeShaCache = HashBasedTable.create();
Map<Project, ObjectId> subtreeShaCache = new HashMap<>();

Expand All @@ -147,25 +146,14 @@ private static boolean worktreeIsCleanCheckout(TreeWalk treeWalk) {
* We cache the Repository for every Project in {@code gitRoots}, and use dynamic programming to populate it.
*/
protected Repository repositoryFor(Project project) throws IOException {
Repository repo = gitRoots.get(project);
File projectGitDir = GitWorkarounds.getDotGitDir(getDir(project));
if (projectGitDir == null || !RepositoryCache.FileKey.isGitRepository(projectGitDir, FS.DETECTED)) {
throw new IllegalArgumentException("Cannot find git repository in any parent directory");
}
Repository repo = gitRoots.get(projectGitDir);
if (repo == null) {
if (isGitRoot(getDir(project))) {
repo = createRepo(getDir(project));
} else {
Project parentProj = getParent(project);
if (parentProj == null) {
repo = traverseParentsUntil(getDir(project).getParentFile(), null);
if (repo == null) {
throw new IllegalArgumentException("Cannot find git repository in any parent directory");
}
} else {
repo = traverseParentsUntil(getDir(project).getParentFile(), getDir(parentProj));
if (repo == null) {
repo = repositoryFor(parentProj);
}
}
}
gitRoots.put(project, repo);
repo = FileRepositoryBuilder.create(projectGitDir);
gitRoots.put(projectGitDir, repo);
}
return repo;
}
Expand All @@ -174,26 +162,6 @@ protected Repository repositoryFor(Project project) throws IOException {

protected abstract @Nullable Project getParent(Project project);

private static @Nullable Repository traverseParentsUntil(File startWith, @Nullable File file) throws IOException {
while (startWith != null && !Objects.equals(startWith, file)) {
if (isGitRoot(startWith)) {
return createRepo(startWith);
} else {
startWith = startWith.getParentFile();
}
}
return null;
}

private static boolean isGitRoot(File dir) {
File dotGit = GitWorkarounds.getDotGitDir(dir);
return dotGit != null && RepositoryCache.FileKey.isGitRepository(dotGit, FS.DETECTED);
}

static Repository createRepo(File dir) throws IOException {
return FileRepositoryBuilder.create(GitWorkarounds.getDotGitDir(dir));
}

/**
* Fast way to return treeSha of the given ref against the git repository which stores the given project.
* Because of parallel project evaluation, there may be races here, so we synchronize on ourselves. However, this method
Expand Down
2 changes: 2 additions & 0 deletions plugin-gradle/CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
We adhere to the [keepachangelog](https://keepachangelog.com/en/1.0.0/) format (starting after version `3.27.0`).

## [Unreleased]
### Fixed
* Improve memory usage when using git ratchet ([#1426](https://github.com/diffplug/spotless/pull/1426))

## [6.12.0] - 2022-11-24
### Added
Expand Down
3 changes: 3 additions & 0 deletions plugin-maven/CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,16 @@
We adhere to the [keepachangelog](https://keepachangelog.com/en/1.0.0/) format (starting after version `1.27.0`).

## [Unreleased]
### Fixed
* Improve memory usage when using git ratchet ([#1426](https://github.com/diffplug/spotless/pull/1426))

## [2.28.0] - 2022-11-24
### Added
* `importOrder` now support groups of imports without blank lines ([#1401](https://github.com/diffplug/spotless/pull/1401))
### Fixed
* Don't treat `@Value` as a type annotation [#1367](https://github.com/diffplug/spotless/pull/1367)
* Support `ktlint_disabled_rules` in `ktlint` 0.47.x [#1378](https://github.com/diffplug/spotless/pull/1378)
* Share git repositories across projects when using ratchet ([#1426](https://github.com/diffplug/spotless/pull/1426))
### Changes
* Bump default `ktfmt` version to latest `0.40` -> `0.41` ([#1340](https://github.com/diffplug/spotless/pull/1340))
* Bump default `scalafmt` version to latest `3.5.9` -> `3.6.1` ([#1373](https://github.com/diffplug/spotless/pull/1373))
Expand Down