Skip to content

issues Search Results · repo:lucidrains/memory-efficient-attention-pytorch language:Python

Filter by

6 results
 (73 ms)

6 results

inlucidrains/memory-efficient-attention-pytorch (press backspace or delete to remove)

lucidrains/memory-efficient-attention-pytorch
Example for long embedding sequence prediction using this attention

Hi, I am working on long embedding vector sequence problem where embedding size is 500k and I want to predict multistep ahead like 100 time steps. I followed gpt2 type architecture but I got memory problem ...
  • ramdhan1989
  • Opened 
    on Jun 3, 2024
  • #8

lucidrains/memory-efficient-attention-pytorch
Making this work with relative position bias from XTransformers

Is there a way to make this work with RelativePositionBias. Currently this produces an attention bias of size $BHN^2$ where B is batch size, H is number of heads and N is input size. Can this be chunked ...
  • pfeatherstone
  • 5
  • Opened 
    on Dec 2, 2022
  • #5

lucidrains/memory-efficient-attention-pytorch
save_for_backward can only save variables, but argument 5 is of type bool

Hi, Thank you for your indescribable work. I was trying to test your method specifically for cross-attention but It seems I get the error save_for_backward can only save variables, but argument 5 is of ...
  • abalikhan
  • 1
  • Opened 
    on Oct 26, 2022
  • #4

lucidrains/memory-efficient-attention-pytorch
Checkpointing is not compatible with .grad() or when an `inputs` parameter is passed to .backward()

https://github.com/lucidrains/memory-efficient-attention-pytorch/blob/35559a05572f9d4eb982a8e2e399b40a2d61b85c/memory_efficient_attention_pytorch/memory_efficient_attention.py#L95 Should this be: summarize_qkv_fn ...
  • vrobot
  • 1
  • Opened 
    on Oct 5, 2022
  • #3

lucidrains/memory-efficient-attention-pytorch
[feature request] Combining with flash attention?

There is a new algorithm to optimize the qkv attention, https://github.com/HazyResearch/flash-attention https://arxiv.org/abs/2205.14135 It optimises the qkv attention part. Maybe you can look into integrating ...
  • Vbansal21
  • 15
  • Opened 
    on Jun 3, 2022
  • #2

lucidrains/memory-efficient-attention-pytorch
i did this, we could build on top

Hi there! It seems I did already some of the code... https://github.com/CHARM-Tx/linear_mem_attention_pytorch could we build on top of this? I talked to https://github.com/Chillee about an experimental ...
  • hypnopump
  • 5
  • Opened 
    on Mar 3, 2022
  • #1
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue search results · GitHub