Alternatives and similar repositories for tokenizer:

Users that are interested in tokenizer are comparing it to the libraries listed below

• mast-group / OpenVocabCodeNLM
  Contains the code for our ICSE 2020 paper: Big Code != Big Vocabulary: Open-Vocabulary Language Models for Source Code and for its earlie…
  ☆83Updated last year
• shramos / pyc-cfg
  Pyc-cfg is a pure python control flow graph builder for almost all Ansi C programming language.
  ☆51Updated 7 years ago
• ASSERT-KTH / sequencer
  Sequence-to-Sequence Learning for End-to-End Program Repair (IEEE TSE 2019). Open-science repo. http://arxiv.org/pdf/1901.01808
  ☆82Updated last year
• giganticode / codeprep
  A toolkit for pre-processing large source code corpora
  ☆47Updated 2 years ago
• srcML / srcML
  srcML Toolkit
  ☆128Updated this week
• sh1doy / summarization_tf
  Implementation of "Automatic Source Code Summarization with Extended Tree-LSTM"
  ☆36Updated 2 years ago
• mast-group / codemining-core
  A set of tools for extracting tokens and ASTs from code
  ☆22Updated 6 years ago
• mast-group / api-mining
  Probabilistic API Mining
  ☆53Updated 7 years ago
• src-d / code2vec
  MLonCode community effort to implement Learning Distributed Representations of Code (https://arxiv.org/pdf/1803.09473.pdf)
  ☆39Updated 6 years ago
• clonebench / BigCloneBench
  ☆111Updated 2 years ago
• guxd / deepAPI
  Repository for Deep API Learning (DeepAPI)
  ☆54Updated 3 years ago
• Kolkir / cppminer
  cppminer produces a code2seq compatible datasets from C++ code bases.
  ☆23Updated 4 years ago
• parasol-aser / deepsim
  Deep learning code semantic similarity
  ☆60Updated 5 years ago
• mwcvitkovic / Open-Vocabulary-Learning-on-Source-Code-with-a-Graph-Structured-Cache--Code-Preprocessor
  Library for preprocessing java source code into Augmented ASTs, as per the paper Open Vocabulary Learning on Source Code with a Graph-Str…
  ☆21Updated 6 years ago
• bentrevett / code2vec
  A PyTorch implementation of `code2vec: Learning Distributed Representations of Code` (Alon et al., 2018)
  ☆35Updated 5 years ago
• JetBrains-Research / code2seq
  PyTorch's implementation of the code2seq model.
  ☆61Updated 6 months ago
• SLP-team / SLP-Core
  Your library for dynamic language modeling
  ☆67Updated 6 years ago
• micheletufano / AutoenCODE
  AutoenCODE is a Deep Learning infrastructure that allows to encode source code fragments into vector representations, which can be used t…
  ☆61Updated 6 years ago
• hvdthong / PatchNetTool
  ☆25Updated 4 years ago
• tianyi-zhang / ExampleStack-ICSE-Artifact
  ☆16Updated 7 months ago
• TruX-DTF / AVATAR
  AVATAR: Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations
  ☆26Updated 3 years ago
• tushartushar / DeepLearningSmells
  Smelling smells using Deep Learning
  ☆43Updated 3 years ago
• bentrevett / extreme-summarization-of-source-code
  Implementation of 'A Convolutional Attention Network for Extreme Summarization of Source Code' in PyTorch using TorchText
  ☆13Updated 6 years ago
• CUHK-ARISE / ml4code-dataset
  A collection of datasets for machine learning for big code
  ☆51Updated 3 years ago
• sonoisa / code2vec
  an implementation of "code2vec: Learning Distributed Representations of Code"
  ☆30Updated 7 months ago
• danhper / bigcode-tools
  Set of tools to help working with "Big Code"
  ☆43Updated 2 years ago
• squaresLab / ManyBugs
  ☆16Updated 5 years ago
• Jur1cek / gcj-dataset
  Collected solutions from Google Code Jam programming competition (2008-2020).
  ☆63Updated 5 months ago
• bdqnghi / ast-node-encoding
  A tool to convert nodes in an Abstract Syntax Tree into vector embeddings
  ☆74Updated 2 years ago
• skyhover / Deckard
  Code clone detection; clone-related bug detection; sematic clone analysis
  ☆213Updated 11 months ago