This project implements a distributed pipeline for NLP model training using Apache Spark and DeepLearning4J (DL4J). The methodology utilizes a sliding window approach for data preparation, positional embeddings for token encoding, and Word2Vec model training with parallel processing. The model and training process is designed for scalability and op
apache-spark tensorflow word2vec scalatest dl4j deeplearning4j sliding-window llm llm-training logback-classic mapreduce-scala
-
Updated
Nov 17, 2024 - Scala