[SYCL] Implement eviction in persistent cache (#16289)

This PR implements eviction for persistent cache.
Eviction is disabled by default and can be controlled by the user via
`SYCL_CACHE_MAX_SIZE` variable.

Here's how eviction works:
1. File size: A file called, `cache_size.txt`, is stored at the root of
persistent cache. Every time a process adds a new entry to cache, it
will also update the `cache_size.txt` file. This file is used to track
the size of persistent cache. For backwards compatibility, if SYCL RT
does not find `cache_size.txt` in the cache root, it will create once.
All access to `cache_size.txt` are done using `LockCacheItem`, to
prevent data races.
2. When adding a new entry to cache, SYCL RT will check the
`cache_size.txt` file and if the cache size exceeds the threshold,
eviction is triggered.
3. When a cache entry is created/accessed, SYCL RT create a file in that
cache entry to store the access time. This file is later read during
eviction.
4. During eviction, SYCL RT will determine the last access time of each
cache entry and items are evicted based on the LRU policy.

Author: uditagarwal97

sycl/doc/design/KernelProgramCache.md

@@ -415,15 +415,16 @@ When adding a new program to cache, we check if the size of the program cache ex
 
 #### Persistent cache eviction
 
-Persistent cache eviction is going to be applied based on file last access
-(read/write) date (access time). On SYCL application shutdown phase cache
-eviction process is initiated which walks through cache directories as follows:
-
-- if the file is locked, go to the next file;
-- otherwise check file access time:
-  - if file access time is above threshold, delete the file and remove parent
-    directory while they are unlocked and empty;
-  - otherwise do nothing.
+Persistent cache eviction can be enabled using the SYCL_CACHE_MAX_SIZE environment variable and is based on the LRU strategy.
+
+- A new file, called `cache_size.txt`, is created at the root of the persistent cache directory. This file contains the total size of the cache in bytes. When a new item is added to the cache, the size of the item is added to the total size in the `cache_size.txt` file. When the total size exceeds the threshold, the eviction process is initiated.
+
+- Whenever a cache entry is added or accessed, the corresponding cache item directory is updated with the current time. This is done by creating a new file, called `<entry name>_access_time.txt`, in the cache item directory. This file contains the current time in nanoseconds since the epoch. When the eviction process is initiated, we use this file to determine the last access time of the cache item.
+
+- When a new item is added to the cache, we check if the total size exceeds the threshold. If so, we iterate through the cache item directories and delete the least recently accessed items until the total size is below half the cache size.
+
+Note that once the eviction is triggered, the cache size is reduced to half the cache size to avoid frequent eviction.
+
 
 ## Cache limitations
 

sycl/include/sycl/detail/os_util.hpp

@@ -12,7 +12,8 @@
 
 #include <sycl/detail/export.hpp> // for __SYCL_EXPORT
 
-#include <cstdlib>    // for size_t
+#include <cstdlib> // for size_t
+#include <functional>
 #include <string>     // for string
 #include <sys/stat.h> // for stat
 
@@ -90,6 +91,20 @@ class __SYCL_EXPORT OSUtil {
   }
 };
 
+// These functions are not a part of OSUtils class to prevent
+// exporting them as ABI. They are only used in persistent cache
+// implementation and should not be exposed to the end users.
+// Get size of directory in bytes.
+size_t getDirectorySize(const std::string &Path);
+
+// Get size of file in bytes.
+size_t getFileSize(const std::string &Path);
+
+// Function to recursively iterate over the directory and execute
+// 'Func' on each regular file.
+void fileTreeWalk(const std::string Path,
+                  std::function<void(const std::string)> Func);
+
 } // namespace detail
 } // namespace _V1
 } // namespace sycl

sycl/source/detail/config.hpp

@@ -806,6 +806,56 @@ template <> class SYCLConfig<SYCL_IN_MEM_CACHE_EVICTION_THRESHOLD> {
   }
 };
 
+// SYCL_CACHE_MAX_SIZE accepts an integer that specifies
+// the maximum size of the on-disk Program cache.
+// Cache eviction is performed when the cache size exceeds the threshold.
+// The thresholds are specified in bytes.
+// The default value is "0" which means that eviction is disabled.
+template <> class SYCLConfig<SYCL_CACHE_MAX_SIZE> {
+  using BaseT = SYCLConfigBase<SYCL_CACHE_MAX_SIZE>;
+
+public:
+  static long long get() { return getCachedValue(); }
+  static void reset() { (void)getCachedValue(true); }
+
+  static long long getProgramCacheSize() { return getCachedValue(); }
+
+  static bool isPersistentCacheEvictionEnabled() {
+    return getProgramCacheSize() > 0;
+  }
+
+private:
+  static long long getCachedValue(bool ResetCache = false) {
+    const auto Parser = []() {
+      const char *ValStr = BaseT::getRawValue();
+
+      // Disable eviction by default.
+      if (!ValStr)
+        return (long long)0;
+
+      long long CacheSize = 0;
+      try {
+        CacheSize = std::stoll(ValStr);
+        if (CacheSize < 0)
+          throw INVALID_CONFIG_EXCEPTION(BaseT, "Value must be non-negative");
+      } catch (...) {
+        std::string Msg =
+            std::string{"Invalid input to SYCL_CACHE_MAX_SIZE. Please try "
+                        "a positive integer."};
+        throw exception(make_error_code(errc::runtime), Msg);
+      }
+
+      return CacheSize;
+    };
+
+    static auto EvictionThresholds = Parser();
+    if (ResetCache)
+      EvictionThresholds = Parser();
+
+    return EvictionThresholds;
+  }
+};
+
 #undef INVALID_CONFIG_EXCEPTION
 
 } // namespace detail

sycl/source/detail/os_util.cpp

@@ -7,14 +7,20 @@
 //===----------------------------------------------------------------------===//
 
 #include <sycl/detail/os_util.hpp>
+#include <sycl/exception.hpp>
 
 #include <cassert>
 #include <limits>
 
-#if __GNUC__ && __GNUC__ < 8
-// Don't include <filesystem> for GCC versions less than 8
+// For GCC versions less than 8, use experimental/filesystem.
+#if defined(__has_include) && __has_include(<filesystem>)
+#include <filesystem>
+namespace fs = std::filesystem;
+#elif defined(__has_include) && __has_include(<experimental/filesystem>)
+#include <experimental/filesystem>
+namespace fs = std::experimental::filesystem;
 #else
-#include <filesystem> // C++ 17 std::create_directories
+#error "OSUtils requires C++ filesystem support"
 #endif
 
 #if defined(__SYCL_RT_OS_LINUX)
@@ -277,6 +283,45 @@ int OSUtil::makeDir(const char *Dir) {
   return 0;
 }
 
+// Get size of file in bytes.
+size_t getFileSize(const std::string &Path) {
+  return static_cast<size_t>(fs::file_size(Path));
+}
+
+// Function to recursively iterate over the directory and execute
+// 'Func' on each regular file.
+void fileTreeWalk(const std::string Path,
+                  std::function<void(const std::string)> Func) {
+
+  std::error_code EC;
+  for (auto It = fs::recursive_directory_iterator(Path, EC);
+       It != fs::recursive_directory_iterator(); It.increment(EC)) {
+
+    // Errors can happen if a file was removed/added during the iteration.
+    if (EC)
+      throw sycl::exception(
+          make_error_code(errc::runtime),
+          "Failed to do File Tree Walk. Ensure that the directory is not "
+          "getting updated while FileTreeWalk is in progress.: " +
+              Path + "\n" + EC.message());
+
+    if (fs::is_regular_file(It->path()))
+      Func(It->path().string());
+  }
+}
+
+// Get size of a directory in bytes.
+size_t getDirectorySize(const std::string &Path) {
+  size_t DirSizeVar = 0;
+
+  auto CollectFIleSize = [&DirSizeVar](const std::string Path) {
+    DirSizeVar += getFileSize(Path);
+  };
+  fileTreeWalk(Path, CollectFIleSize);
+
+  return DirSizeVar;
+}
+
 } // namespace detail
 } // namespace _V1
 } // namespace sycl