how to use pl to process tfrecords data?

[jaffe-fly]

how to use LightningDataModule to process tfrecords data, anyone can give a Tutorial

[tshu-w]

How you use python and pytorch to handle tfrecords data is how you use it in LightningDataModule. pytorchlightning is just a wrapper.

[jaffe-fly]

i know this . so how to process tfrecords with pytorch efficently

[tshu-w]

... You should ask Google, are the instructions for this package not clear? https://github.com/vahidk/tfrecord#usage

[Borda]

Depends on what data you have inside TFRecord, but you can see usage:
https://github.com/vahidk/tfrecord#reading-tfexample-records-in-pytorch

def train_dataloader():
    # index_path = None
    # tfrecord_path = "/tmp/data.tfrecord"
    # description = {"image": "byte", "label": "float"}
    dataset = TFRecordDataset(tfrecord_path, index_path, description)
    loader = torch.utils.data.DataLoader(dataset, batch_size=32)
    return loader

also check how to work with dataset - https://discuss.pytorch.org/t/read-dataset-from-tfrecord-format/16409/15