A shortcut action to cache Julia artifacts, packages, and registries.
  • TypeScript 90.4%
  • Julia 9.6%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
William Moses 9353db09a0
Unpack a GCS restore into a staging directory so a failed stream leaves the depot untouched (#264)
* Discard a GCS restore that fails part-way, and never re-save it

The GCS restore streams the archive straight through zstd and tar into the
depot. If the stream breaks (a read ECONNRESET from GCS, say), whatever tar
had written stays in place, including the file it was in the middle of
writing, cut short. streamRestore rejects, but run() only logs a warning
and carries on, so the job runs against a depot holding truncated files.
The post step then saves that depot: save-always defaults to true, and the
save also copies the archive over the restore-key object, which every job
that shares the restore key reads next. One dropped connection thus turns
into a cache that every following job restores and re-saves.

That is what happened in EnzymeAD/Enzyme-JAX: one job logged
`Failed to restore cache from GCS: read ECONNRESET`, came up with 80M of
scratchspaces instead of 652M (a 216M libtpu.so among them), crashed with
SIGBUS when dlopen mapped the truncated library, and saved. Every TPU job
on every branch bus-errored the same way for the next ten days.

Two changes. When the restore stream fails, remove the cache paths the
restore created (paths that existed beforehand are kept, with a warning,
since removing them would take the user's own files too) and record that
the restore failed. In the post step, skip the save when the restore
failed, whatever save-always says: there is nothing trustworthy to save.
The successful-restore log line and cache-matched-key now come after the
stream completes rather than before it starts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRuYsAfRi5TvpWuAQeG2b

* Rebuild dist

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRuYsAfRi5TvpWuAQeG2b

* Remove exactly what a failed GCS restore extracted, and name it as GCS-only

Review follow-ups:

- Have tar list what it extracts (GNU tar on stdout, bsdtar on stderr)
  and remove exactly those entries when the stream fails, instead of
  removing whole cache paths that did not exist beforehand. A cache path
  such as ~/.julia/artifacts usually exists before the restore, and the
  old approach would then have left truncated files under it. Files are
  removed first, a directory only once it is empty, and cache paths that
  existed beforehand are kept even when empty.
- Rename the helper to discardPartialGcsRestore, since only the Google
  Cloud Storage pathway uses it, and the state to restore-failed-partway
  so it is not read as a cache miss.
- Shorter comments, and the README paragraph is marked Google
  Cloud-specific and moved below the v3 note.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Rebuild dist

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Name streamRestore after the Google Cloud Storage pathway that uses it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Rebuild dist

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Keep tar's listing as raw output and parse it only when the restore fails

Building one string per extracted entry as the listing arrived retained
about 93 MB of heap for a 100k-file depot, since V8 keeps sliced strings
alive through their parents. Keep the chunks tar writes instead, about
3.5 MB for that depot, and turn them into paths only when there is
something to remove.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Rebuild dist

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Unpack a GCS restore into a staging directory and move it into place on success

Rather than listing what tar extracts and removing it when the stream
breaks, unpack the archive into a staging directory created inside the
depot (so it is on the same filesystem) and only move the cache paths
into place once the whole archive has arrived, merging into directories
that already exist and replacing files that do. A restore that fails
part-way then leaves the depot exactly as it was, so a truncated file
can never reach the depot, and the post step no longer needs to know
whether the restore failed: post.ts is back to what it was on main.

Restoring a 100k-file, 831 MB archive from memory to local disk takes
the same 2.9 s as before this PR.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Rebuild dist

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb

* Harden the GCS staging restore

- Settle streamGcsRestore only once the download, the decompressor and
  tar have all finished, in whatever order. tar exits 0 on an archive cut
  at a member boundary, so its exit alone did not show the whole archive
  arrived: with a decompressor that failed after tar exited, the old code
  resolved and moved a partial tree into place. Using pipeline() also
  handles EPIPE on the child stdin streams, which was an uncaught
  exception, and waiting for both children before settling stops the
  staging directory being removed while tar still writes into it.
- Restore into the target of a cache path that is a symlink to a
  directory instead of replacing the symlink, and copy when a rename
  crosses filesystems (EXDEV).
- Name the cache path in the error when moving into place fails; with
  #268 the post step then does not save the partially restored depot.
- Warn, and do not report a match, when the archive contains none of
  the cache paths (e.g. it was saved with a different depot path).
- Make removing the staging directory best-effort, so a failure there
  is not reported as a failed restore, and remove staging directories
  older than 6 hours left behind by killed jobs. The age guard keeps a
  restore in progress in a depot shared between runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011zZQBXE6FdVTPHuEcqu1VC

* Rebuild dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011zZQBXE6FdVTPHuEcqu1VC

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Mosè Giordano <mose@gnu.org>
2026-10-05 17:46:28 -04:00
.github CI: deepen the General registry clone before resetting it 20 commits back (#265) 2026-09-27 21:48:06 -04:00
devdocs Devdocs: document the process for making a new release of this action (#128) 2024-05-22 23:57:30 -04:00
dist Unpack a GCS restore into a staging directory so a failed stream leaves the depot untouched (#264) 2026-10-05 17:46:28 -04:00
src Unpack a GCS restore into a staging directory so a failed stream leaves the depot untouched (#264) 2026-10-05 17:46:28 -04:00
.gitattributes .gitattributes: Hide generated JS code from diffs and language stats (#253) 2026-07-30 21:02:47 -04:00
.gitignore Javascript version of existing action which also saves if job fails (#198) 2026-03-07 08:50:45 -05:00
.npmrc Set the NPM cooldown in the .npmrc file for local usage (#234) 2026-05-28 14:37:52 -04:00
action.yml Cache the objcache on 1.14+ (#263) 2026-09-13 14:14:40 +01:00
AGENTS.md AGENTS.md: Add rules about putting changes to generated files (dist/* and package-lock.json) in separate commits (#224) 2026-05-23 07:45:19 -04:00
CLAUDE.md Add instructions for LLM coding agents (AGENTS.md and CLAUDE.md) (#206) 2026-03-18 16:51:57 -04:00
handle_caches.jl Add Google Cloud Storage cache support (#204) 2026-04-30 22:05:34 -04:00
LICENSE Initial commit 2021-12-26 15:20:58 +01:00
package-lock.json Bump @vercel/ncc from 0.44.1 to 0.45.0 (#261) 2026-10-01 10:37:31 +01:00
package.json Bump @vercel/ncc from 0.44.1 to 0.45.0 (#261) 2026-10-01 10:37:31 +01:00
README.md Unpack a GCS restore into a staging directory so a failed stream leaves the depot untouched (#264) 2026-10-05 17:46:28 -04:00
tsconfig.json Migrate from JavaScript to TypeScript (#248) 2026-08-15 01:40:00 -04:00

julia-actions/cache Action

A shortcut action to cache Julia depot contents to reduce GitHub Actions running time.

Usage

An example workflow that uses this action might look like this:

name: CI

on: [push, pull_request]

# needed to allow julia-actions/cache to delete old caches that it has created
permissions:
  actions: write
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v6
    - uses: julia-actions/setup-julia@v3
    - uses: julia-actions/cache@v3
    - uses: julia-actions/julia-buildpkg@v1
    - uses: julia-actions/julia-runtest@v1

By default all depot directories called out below are cached.

Optional Inputs

  • cache-name - The cache key prefix. Defaults to julia-cache;workflow=${{ github.workflow }};job=${{ github.job }}. The key body automatically includes the OS and, unless disabled with include-matrix, the matrix vars. Include any other parameters/details in this prefix to ensure one unique cache key per concurrent job type.
  • include-matrix - Whether to include the matrix values when constructing the cache key. Defaults to true.
  • depot - Path to a Julia depot directory where cached data will be saved to and restored from. Defaults to the first depot in JULIA_DEPOT_PATH if specified. Otherwise, defaults to ~/.julia.
  • cache-artifacts - Whether to cache the depot's artifacts directory. Defaults to true.
  • cache-packages - Whether to cache the depot's packages directory. Defaults to true.
  • cache-registries - Whether to cache the depot's registries directory. Defaults to true.
  • cache-compiled - Whether to cache the depot's compiled directory. Defaults to true.
  • cache-scratchspaces - Whether to cache the depot's scratchspaces directory. Defaults to true.
  • cache-logs - Whether to cache the depot's logs directory. Defaults to true. Helps auto-Pkg.gc() keep the cache small.
  • cache-objcache - Whether to cache the depot's cache directory. Defaults to true. On Julia 1.14 and later this holds the LLVM object cache (objcache), which stores native code for compiled LLVM IR so it does not have to be regenerated in later runs. Its size is bounded by Julia (512 MB by default, see JULIA_OBJCACHE_CAPACITY).
  • save-always - Whether to save the cache even if the job fails. Defaults to true.
  • delete-old-caches - Whether to delete old caches for the given key. Defaults to true.
  • token - A GitHub PAT. Defaults to github.token. Requires repo scope to enable the deletion of old caches.

Outputs

  • cache-hit - A boolean value to indicate an exact match was found for the primary key. Returns "" when the key is new. Forwarded from actions/cache.
  • cache-paths - A list of paths (as a newline-separated string) that were cached.
  • cache-key - The cache key that was used for this run.

How It Works

This action is a wrapper around https://github.com/actions/cache. In summary, this action stores the files in the aforementioned paths in one compressed file when running for the first time. This cached file is then restored upon the second run, and afterwards resaved under a new key, and the previous cache deleted. The benefit of caching is that downloading one big file is quicker than downloading many different files from many different locations and precompiling them.

By default, this action removes caches that were previously made by jobs on the same branch with the same restore key. GitHub automatically removes old caches after a certain period or when the repository cache allocation is full. It is, however, more efficient to explicitly remove old caches to improve caching for less frequently run jobs.

For more information about GitHub caching generically, for example how to manually delete caches, see this GitHub documentation page.

Cache keys

The cache key that the cache will be saved as is based on:

  • The cache-name input
  • All variables in the matrix (unless disabled via include-matrix: 'false')
  • The runner.os (may be in the matrix too, but included for safety)
  • The run id
  • The run attempt number

Note

If there is job concurrency that is not fully defined by a matrix you should ensure that cache-name is unique for each concurrent job, otherwise caching may not be effective.

Cache Retention

This action automatically deletes old caches that match the first 4 fields of the above key:

  • The cache-name input
  • All variables in the matrix (unless disabled via include-matrix: 'false')
  • The runner.os (may be in the matrix too, but included for safety)

Which means your caches files will not grow needlessly. GitHub also deletes cache files after 7 days of not being accessed, and there is a limit of 10 GB for the total size of cache files associated to each repository.

Note

To allow deletion of caches you will likely need to grant the following permissions to the GITHUB_TOKEN by adding this to your GitHub actions workflow:

permissions:
  actions: write
  contents: read

(Note this won't work for fork PRs but should once merged) Or provide a token with repo scope via the token input option. See https://cli.github.com/manual/gh_cache_delete

To disable deletion set input delete-old-caches: 'false'.

Caching even if a job step fails

By default, this action saves the cache even if the job fails (save-always: true). This differs from GitHub's own actions/cache but is useful because Julia package installation and precompilation can take a long time, and you don't want to lose that work just because a later step (like tests) failed.

If you prefer the old behavior where caches are only saved on successful jobs, set save-always: false.

Note

This behavior was changed in v3, which switched from a composite action to a JavaScript action to enable proper post-step cache saving regardless of job status. In v2, saving on failure required a manual workaround with actions/cache/save. See https://github.com/julia-actions/cache/pull/169.

Google Cloud-specific: a cache restored from Google Cloud Storage is unpacked into a staging directory inside the depot and only moved into place once the whole archive has arrived, so a restore that fails part-way (for example because the connection drops) leaves the depot as it was. As on the GitHub cache path, a restore that does not complete is never saved at the end of the job, and cached registries are not restored over registries already in the depot.

Cache Garbage Collection

Caches are restored and re-saved after every run, retaining the state of the depot throughout runs. Their size will be regulated like a local depot automatically by the automatic Pkg.gc() functionality that clears out old content, which is made possible because the /log contents are cached.

Third Party Notice

This action is built around actions/cache and includes parts of that action. actions/cache has been released under the following licence:

The MIT License (MIT)

Copyright (c) 2018 GitHub, Inc. and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.