Guided Decoding

[johnml1135]

From the papers out there, determine the best path forward, research and implement guided decoding. Assess the improved Bleu score and user assessment of quality. Address concerns with different types of languages with preixes and suffixes on proper names and key terms.

[johnml1135]

https://huggingface.co/blog/constrained-beam-search
huggingface/transformers#15761

[johnml1135]

We should likely just use the huggingface implementation from 2022 (see links above) - but it may need to be modified. A few reasons:

• It does not use alignment to guide when to add tokens
• It cannot deal with wildcards - which is useful in many languages we deal with - but we may be able to build it on disjunctive constraints

[ddaspit]

I added preliminary support for using HF constrained beam search to silnlp. From the experiments I have run, it doesn't work very well.

[johnml1135]

@ddaspit - do you know why the tests didn't go that well? Do you have the results documented somewhere? Is it "keyterms with asterisks don't work well" or "their algorithm is poor" or "certain languages don't do well with this"? Is it worth more research now, or do we want someone else to lead the charge? Do we need to add alignment data to enhance it? LILT appears to have been able to get this working well enough to integrate into their main offering - so I am inclined to believe it is possible to have it be advantageous.

[ddaspit]

There definitely seems to be something wrong with the implementation in HF. Here is an issue that describes the problems I was seeing.

[johnml1135]

While not implemented, this may do better than the current hugging face implementation: https://arxiv.org/pdf/2112.08726.pdf - with this code: https://github.com/GXimingLu/a_star_neurologic.

[johnml1135]

It appears that the old "constraint" decoding is not supported anymore, but rather sequence_bias and repetition_penalty in the GenerationConfig.

I strongly suspect that this new set of updates may be worth another round of research to see how well it fairs.

[johnml1135]

Actually, I took a look at sequence_bias, and it is a bit different than constrained decoding - potentially more for LLM's than NLLB - still it may be worth another look. When we checked out constrained decoding last, it was verifiably broken, and it is not due to be fixed or maintained by the huggingface team. If we want something that will work today and tomorrow, we should investigate sequence_bias.

[ddaspit]

Sequence bias looks interesting. Hopefully, it works better than constrained decoding. At some point, we should update the existing code to use sequence bias instead.