- TypeScript 90.4%
- Julia 9.6%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
* Discard a GCS restore that fails part-way, and never re-save it The GCS restore streams the archive straight through zstd and tar into the depot. If the stream breaks (a read ECONNRESET from GCS, say), whatever tar had written stays in place, including the file it was in the middle of writing, cut short. streamRestore rejects, but run() only logs a warning and carries on, so the job runs against a depot holding truncated files. The post step then saves that depot: save-always defaults to true, and the save also copies the archive over the restore-key object, which every job that shares the restore key reads next. One dropped connection thus turns into a cache that every following job restores and re-saves. That is what happened in EnzymeAD/Enzyme-JAX: one job logged `Failed to restore cache from GCS: read ECONNRESET`, came up with 80M of scratchspaces instead of 652M (a 216M libtpu.so among them), crashed with SIGBUS when dlopen mapped the truncated library, and saved. Every TPU job on every branch bus-errored the same way for the next ten days. Two changes. When the restore stream fails, remove the cache paths the restore created (paths that existed beforehand are kept, with a warning, since removing them would take the user's own files too) and record that the restore failed. In the post step, skip the save when the restore failed, whatever save-always says: there is nothing trustworthy to save. The successful-restore log line and cache-matched-key now come after the stream completes rather than before it starts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRuYsAfRi5TvpWuAQeG2b * Rebuild dist Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRuYsAfRi5TvpWuAQeG2b * Remove exactly what a failed GCS restore extracted, and name it as GCS-only Review follow-ups: - Have tar list what it extracts (GNU tar on stdout, bsdtar on stderr) and remove exactly those entries when the stream fails, instead of removing whole cache paths that did not exist beforehand. A cache path such as ~/.julia/artifacts usually exists before the restore, and the old approach would then have left truncated files under it. Files are removed first, a directory only once it is empty, and cache paths that existed beforehand are kept even when empty. - Rename the helper to discardPartialGcsRestore, since only the Google Cloud Storage pathway uses it, and the state to restore-failed-partway so it is not read as a cache miss. - Shorter comments, and the README paragraph is marked Google Cloud-specific and moved below the v3 note. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Rebuild dist Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Name streamRestore after the Google Cloud Storage pathway that uses it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Rebuild dist Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Keep tar's listing as raw output and parse it only when the restore fails Building one string per extracted entry as the listing arrived retained about 93 MB of heap for a 100k-file depot, since V8 keeps sliced strings alive through their parents. Keep the chunks tar writes instead, about 3.5 MB for that depot, and turn them into paths only when there is something to remove. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Rebuild dist Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Unpack a GCS restore into a staging directory and move it into place on success Rather than listing what tar extracts and removing it when the stream breaks, unpack the archive into a staging directory created inside the depot (so it is on the same filesystem) and only move the cache paths into place once the whole archive has arrived, merging into directories that already exist and replacing files that do. A restore that fails part-way then leaves the depot exactly as it was, so a truncated file can never reach the depot, and the post step no longer needs to know whether the restore failed: post.ts is back to what it was on main. Restoring a 100k-file, 831 MB archive from memory to local disk takes the same 2.9 s as before this PR. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Rebuild dist Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G84mmXoaait2MBqCH5BKNb * Harden the GCS staging restore - Settle streamGcsRestore only once the download, the decompressor and tar have all finished, in whatever order. tar exits 0 on an archive cut at a member boundary, so its exit alone did not show the whole archive arrived: with a decompressor that failed after tar exited, the old code resolved and moved a partial tree into place. Using pipeline() also handles EPIPE on the child stdin streams, which was an uncaught exception, and waiting for both children before settling stops the staging directory being removed while tar still writes into it. - Restore into the target of a cache path that is a symlink to a directory instead of replacing the symlink, and copy when a rename crosses filesystems (EXDEV). - Name the cache path in the error when moving into place fails; with #268 the post step then does not save the partially restored depot. - Warn, and do not report a match, when the archive contains none of the cache paths (e.g. it was saved with a different depot path). - Make removing the staging directory best-effort, so a failure there is not reported as a failed restore, and remove staging directories older than 6 hours left behind by killed jobs. The age guard keeps a restore in progress in a depot shared between runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011zZQBXE6FdVTPHuEcqu1VC * Rebuild dist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011zZQBXE6FdVTPHuEcqu1VC --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Mosè Giordano <mose@gnu.org> |
||
| .github | ||
| devdocs | ||
| dist | ||
| src | ||
| .gitattributes | ||
| .gitignore | ||
| .npmrc | ||
| action.yml | ||
| AGENTS.md | ||
| CLAUDE.md | ||
| handle_caches.jl | ||
| LICENSE | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
julia-actions/cache Action
A shortcut action to cache Julia depot contents to reduce GitHub Actions running time.
Usage
An example workflow that uses this action might look like this:
name: CI
on: [push, pull_request]
# needed to allow julia-actions/cache to delete old caches that it has created
permissions:
actions: write
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: julia-actions/setup-julia@v3
- uses: julia-actions/cache@v3
- uses: julia-actions/julia-buildpkg@v1
- uses: julia-actions/julia-runtest@v1
By default all depot directories called out below are cached.
Optional Inputs
cache-name- The cache key prefix. Defaults tojulia-cache;workflow=${{ github.workflow }};job=${{ github.job }}. The key body automatically includes the OS and, unless disabled withinclude-matrix, the matrix vars. Include any other parameters/details in this prefix to ensure one unique cache key per concurrent job type.include-matrix- Whether to include the matrix values when constructing the cache key. Defaults totrue.depot- Path to a Julia depot directory where cached data will be saved to and restored from. Defaults to the first depot inJULIA_DEPOT_PATHif specified. Otherwise, defaults to~/.julia.cache-artifacts- Whether to cache the depot'sartifactsdirectory. Defaults totrue.cache-packages- Whether to cache the depot'spackagesdirectory. Defaults totrue.cache-registries- Whether to cache the depot'sregistriesdirectory. Defaults totrue.cache-compiled- Whether to cache the depot'scompileddirectory. Defaults totrue.cache-scratchspaces- Whether to cache the depot'sscratchspacesdirectory. Defaults totrue.cache-logs- Whether to cache the depot'slogsdirectory. Defaults totrue. Helps auto-Pkg.gc()keep the cache small.cache-objcache- Whether to cache the depot'scachedirectory. Defaults totrue. On Julia 1.14 and later this holds the LLVM object cache (objcache), which stores native code for compiled LLVM IR so it does not have to be regenerated in later runs. Its size is bounded by Julia (512 MB by default, seeJULIA_OBJCACHE_CAPACITY).save-always- Whether to save the cache even if the job fails. Defaults totrue.delete-old-caches- Whether to delete old caches for the given key. Defaults totrue.token- A GitHub PAT. Defaults togithub.token. Requiresreposcope to enable the deletion of old caches.
Outputs
cache-hit- A boolean value to indicate an exact match was found for the primary key. Returns "" when the key is new. Forwarded from actions/cache.cache-paths- A list of paths (as a newline-separated string) that were cached.cache-key- The cache key that was used for this run.
How It Works
This action is a wrapper around https://github.com/actions/cache. In summary, this action stores the files in the aforementioned paths in one compressed file when running for the first time. This cached file is then restored upon the second run, and afterwards resaved under a new key, and the previous cache deleted. The benefit of caching is that downloading one big file is quicker than downloading many different files from many different locations and precompiling them.
By default, this action removes caches that were previously made by jobs on the same branch with the same restore key. GitHub automatically removes old caches after a certain period or when the repository cache allocation is full. It is, however, more efficient to explicitly remove old caches to improve caching for less frequently run jobs.
For more information about GitHub caching generically, for example how to manually delete caches, see this GitHub documentation page.
Cache keys
The cache key that the cache will be saved as is based on:
- The
cache-nameinput - All variables in the
matrix(unless disabled viainclude-matrix: 'false') - The
runner.os(may be in the matrix too, but included for safety) - The run id
- The run attempt number
Note
If there is job concurrency that is not fully defined by a matrix you should ensure that
cache-nameis unique for each concurrent job, otherwise caching may not be effective.
Cache Retention
This action automatically deletes old caches that match the first 4 fields of the above key:
- The
cache-nameinput - All variables in the
matrix(unless disabled viainclude-matrix: 'false') - The
runner.os(may be in the matrix too, but included for safety)
Which means your caches files will not grow needlessly. GitHub also deletes cache files after 7 days of not being accessed, and there is a limit of 10 GB for the total size of cache files associated to each repository.
Note
To allow deletion of caches you will likely need to grant the following permissions to the
GITHUB_TOKENby adding this to your GitHub actions workflow:permissions: actions: write contents: read(Note this won't work for fork PRs but should once merged) Or provide a token with
reposcope via thetokeninput option. See https://cli.github.com/manual/gh_cache_delete
To disable deletion set input delete-old-caches: 'false'.
Caching even if a job step fails
By default, this action saves the cache even if the job fails (save-always: true). This differs from GitHub's own actions/cache but is useful because Julia package installation and precompilation can take a long time, and you don't want to lose that work just because a later step (like tests) failed.
If you prefer the old behavior where caches are only saved on successful jobs, set save-always: false.
Note
This behavior was changed in v3, which switched from a composite action to a JavaScript action to enable proper post-step cache saving regardless of job status. In v2, saving on failure required a manual workaround with
actions/cache/save. See https://github.com/julia-actions/cache/pull/169.
Google Cloud-specific: a cache restored from Google Cloud Storage is unpacked into a staging directory inside the depot and only moved into place once the whole archive has arrived, so a restore that fails part-way (for example because the connection drops) leaves the depot as it was. As on the GitHub cache path, a restore that does not complete is never saved at the end of the job, and cached registries are not restored over registries already in the depot.
Cache Garbage Collection
Caches are restored and re-saved after every run, retaining the state of the depot throughout runs.
Their size will be regulated like a local depot automatically by the automatic Pkg.gc() functionality that
clears out old content, which is made possible because the /log contents are cached.
Third Party Notice
This action is built around actions/cache and includes parts of that action. actions/cache has been released under the following licence:
The MIT License (MIT)
Copyright (c) 2018 GitHub, Inc. and contributors
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.