TSNE vis: update the model & embeddings

[bzz]

Some improvements in visualisation relevant to #62.

It's using a 'all-MiniLM-L6-v2' model (87Mb instead of 418Mb, the same 512 context size) that is faster & seems to provide better visualisation.

Before: the previous model

The previous model using both, abstracts and titles.

The new model (batch size 1, abstracts only)

After: the new model + titles + batched

I tied UMAP for it and the results seems less interesting (but didn't experiment much)

UMAP

I also tried using a larger 420Mb model fine-tuned on scientific papers from SciRepEval - allenai/specter2 \w proximity adaptor that takes 1.5min vs 30sec of the above. It can't be switched though the CLI only, as it requires loading an adaptor.

specter2 TSNE

Let me know what you think and which one do you prefer!

[mallamanis]

Thanks for looking into this @bzz! All these options seem quite interesting, yet, it's hard to decide without looking at what each point represents.

May I suggest, that you sample 3-4 papers that you know and try to see which of the visualizations gets "reasonable" neighbors? I'd be happy to go with whatever you find more useful, in that sense 👓

[bzz]

Indeed, that was exactly what I did, I apologize for missing this crucial information :)

Here are the interactive visualisations of embeddings + metadata for all 4 different models, sorted by subjective perception of "cluster quality" that should be easy to explore:

• T-SNE GPT-3, model (text-embedding-ada-002)
• T-SNE sentence-transformers/all-MiniLM-L6-v2, model (new model + titles + batched)
• T-SNE allenai/specter2, model (specter2)
• T-SNE deepset/sentence_bert, model (the previous model)

The best results were achieved with T-SNE hyperparams: ppl 10-20, LR 0.01, 1-2k steps
It's also better to switch Label By to Title.

Clearly identifiable clusters

• Types
• Completion
• Search
• Summarization
• Bugs (localization, fix/repair)
• (OpenAI) Commit messages, Decompilation/Obfuscation, ...

Let me know what you think!

[mallamanis]

This looks great! Thanks a lot for this 💯

[mallamanis]

Hi @bzz it seems like that the Action Fails with this change, I suspect this is due to some restriction on GitHub Actions (memory?). Do you maybe have time to investigate?

https://github.com/ml4code/ml4code.github.io/actions/runs/4952439871

For now, I've reverted this PR in #103

[bzz]

Oh, my! From a quick glance - it may also have to do with some changes in CI runner image actions/runner-images#7188 🤔

I'll setup CI on my fork to try to reproduce and report back.

[bzz]

I suspect this is due to some restriction on GitHub Actions (memory?)

You are right, I missed that the CI has failed repeatedly and it's easily reproducible.

Here is RAM profile across different batch sizes (1 to 512, doubling every ~20sec) 🙄

So the VM gets killed with OOM after exceeding 7Gb RAM limit