Linux worker out of disk space

[victorb]

From @vmx on August 7, 2018 23:36

CI currently fails beause a (some?) Linux worker run out of disk space. See https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs-ipfs-unixfs-engine/detail/PR-223/8/pipeline/22 (and previous build of this job) for more.

Copied from original issue: ipfs-inactive/jenkins#122

[victorb]

From @Stebalien on August 11, 2018 20:43

@victorbjelkholm is there anything the go team can do to get this resolved?

[victorb]

From @magik6k on August 12, 2018 13:47

There is a fix waiting in https://github.com/ipfs/go-ipfs/pull/5053/files#diff-b7ff5824046cf0b1acd30703f283738dR65, but it sort-of depends on js-api tests being stable. I'll open a separate pr for that

[victorb]

From @magik6k on August 12, 2018 13:52

(ipfs/kubo#5374)

[victorb]

Websites are now properly cleaning up after themselves.

We're currently missing golang builds to clean their workspace + cleaning up docker volumes (containers should already be cleaned up now). Otherwise the space is mostly used for golang projects + yarn cache.

[Stebalien]

Shouldn't jenkins be cleaning up after builds?

[victorb]

@Stebalien by default, Jenkins keeps the workspaces across builds.

[victorb]

Several mitigations against this now that should help us. Metrics from all workers regarding disk space (and many other metrics) for alerting and job that runs each day to clean-up. Also made the pipelines better to cleanup after themselves, so we should not see this anymore.