Skip to content

Only process entity data for genPOI to improve performance #1224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
markfickett opened this issue Mar 21, 2015 · 0 comments
Open

Only process entity data for genPOI to improve performance #1224

markfickett opened this issue Mar 21, 2015 · 0 comments

Comments

@markfickett
Copy link
Contributor

I've been profiling genPOI performance (on a single-threaded machine). One of the slow parts is reading/parsing data out of the chunks; I believe much of that data is unnecessary for genPOI.

Profiling result:

genPOI 209.85s (partial)
  entities 209.84s (partial)
    chunk 4616 * 0.00s = 9.93s (max 0.09s)
      get_chunk 4616 * 0.00s = 9.78s (max 0.09s)
        load chunk from region 4616 * 0.00s = 4.66s (max 0.08s)
          create NBTFileReader 4616 * 0.00s = 4.43s (max 0.08s)
            create StringIO for NBT 4617 * 0.00s = 0.04s (max 0.01s)
            zlib decompress 4616 * 0.00s = 1.23s (max 0.01s)
            remainder 0.68s
          seek 4616 * 0.00s = 0.03s (max 0.00s)
        biomes 4616 * 0.00s = 0.06s (max 0.01s)
        blocks 24181 * 0.00s = 0.43s (max 0.01s)
        skylight 24181 * 0.00s = 1.22s (max 0.01s)
        blocklight 24181 * 0.00s = 1.17s (max 0.01s)
        data 24181 * 0.00s = 1.15s (max 0.01s)
      iterate over chunk 4616 * 0.00s = 0.04s (max 0.00s)
    remainder 1.40s

Note especially that reading blocks, skylight, and blocklight contribute nontrivially to get_chunk, but only the result of data is used.

I'll prepare a pull request that adds an option to get_chunk to only read data. In my testing that reduces genPOI time by about a third, from about 300s to about 200s.

The other good optimization candidate looks like parsing data out of the NBTFile, which does not look like as easy a win. In that case I assume a C implementation would be faster; and the same approach as here (loading only required data) would probably be an improvement but the approach is not as clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants