Jenkins keeps running out of Space

[daviddias]

What is the time sink?

Jenkins keeps running out of Space

How do you think it can be solved?

Get bigger machine or develop a post CI task that cleans after

How many hours per week do you waste thanks to this time sink?

Sum of hours per multiple contributor, easily 40 or more.

How important do you think this is to fix?

It is a big PITA

[Stebalien]

Note: part of this was caused by go-ipfs not cleaning up after itself: #118, #70. We've improved the situation but I'm not sure if it's fixed.

[daviddias]

everyday ipfs-inactive/js-ipfs-http-client#806 (comment)

[daviddias]

Duplicate of #118, closing

[alanshaw]

Coincidentally I was just separately saying to @victorbjelkholm that any reason that CI fails because of something other than the tests failing is really bad. It’s super frustrating and time consuming for the developer and wastes resources.

Off the top of my head the big issues are:

• out of disk space errors
• failed npm module installs
• random segfaults

It also really sucks when this happens for contributors as they can’t just restart a build. The fact that this effects the community as well as PL is really bad.

[victorb]

These are the current open issues for addressing these problems:

• Missing packages in registry
  • Custom npm registry running with jenkins master #51 Add a cache layer for npm registry on CI
• Disk space
  • Setup node_exporter + prometheus instance for metrics collection from workers #120 Monitoring of disk space
  • Linux worker out of disk space #118 Collection of potential causes of disk space problems
• Segfaults
  • macOS Node.js 9.2.0 segfaults a lot #112
    • Havent found the reason why, switching nodejs version seems to have stopped it from happening