Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• mast-group / OpenVocabCodeNLM
  Contains the code for our ICSE 2020 paper: Big Code != Big Vocabulary: Open-Vocabulary Language Models for Source Code and for its earlie…
  ☆83Updated 2 years ago
• src-d / code2vec
  MLonCode community effort to implement Learning Distributed Representations of Code (https://arxiv.org/pdf/1803.09473.pdf)
  ☆39Updated 6 years ago
• jjhenkel / code-vectors-artifact
  Artifacts and other data for "Code Vectors: Understanding Programs Through Embedded Abstraced Symbolic Traces"
  ☆22Updated 4 years ago
• ASSERT-KTH / sequencer
  Sequence-to-Sequence Learning for End-to-End Program Repair (IEEE TSE 2019). Open-science repo. http://arxiv.org/pdf/1901.01808
  ☆85Updated last year
• shramos / pyc-cfg
  Pyc-cfg is a pure python control flow graph builder for almost all Ansi C programming language.
  ☆53Updated 7 years ago
• ali-ghanbari / prapr-sc
  Official repository for PraPR source code
  ☆14Updated 4 years ago
• sh1doy / summarization_tf
  Implementation of "Automatic Source Code Summarization with Extended Tree-LSTM"
  ☆36Updated 2 years ago
• mjc92 / buffer_overrun_memory_networks
  ☆20Updated 8 years ago
• pcyin / dire
  Neural Variable Renaming for Decompiled Binaries
  ☆44Updated 5 years ago
• stg-tud / MUBench
  ☆56Updated last year
• sonoisa / code2vec
  an implementation of "code2vec: Learning Distributed Representations of Code"
  ☆30Updated 10 months ago
• sola-st / SemSeed
  ☆25Updated 3 years ago
• micheletufano / AutoenCODE
  AutoenCODE is a Deep Learning infrastructure that allows to encode source code fragments into vector representations, which can be used t…
  ☆60Updated 7 years ago
• srcML / srcML
  srcML Toolkit
  ☆135Updated this week
• hvdthong / PatchNetTool
  ☆26Updated 4 years ago
• Kolkir / cppminer
  cppminer produces a code2seq compatible datasets from C++ code bases.
  ☆23Updated 5 years ago
• clonebench / BigCloneBench
  ☆112Updated 2 years ago
• OOPSLA-2019-BugDetection / OOPSLA-2019-BugDetection
  ☆45Updated 2 years ago
• giganticode / codeprep
  A toolkit for pre-processing large source code corpora
  ☆47Updated 2 years ago
• bstee615 / tree-climber
  Program analysis tools built on tree-sitter (https://github.com/tree-sitter/tree-sitter).
  ☆40Updated last year
• SLP-team / SLP-Core
  Your library for dynamic language modeling
  ☆67Updated 6 years ago
• VHellendoorn / ICLR20-Great
  Data and Code for Reproducing "Global Relational Models of Source Code"
  ☆84Updated 4 years ago
• guxd / deepAPI
  Repository for Deep API Learning (DeepAPI)
  ☆54Updated 3 years ago
• sfzhou5678 / Lancer
  Code for paper "Lancer: Your Code Tell Me What You Need"
  ☆11Updated 2 years ago
• micheletufano / NeuralCodeTranslator
  Neural Code Translator provides instructions, datasets, and a deep learning infrastructure (based on seq2seq) that aims at learning code …
  ☆38Updated 6 years ago
• CUHK-ARISE / ml4code-dataset
  A collection of datasets for machine learning for big code
  ☆56Updated 3 years ago
• squaresLab / ManyBugs
  ☆16Updated 5 years ago
• danhper / bigcode-tools
  Set of tools to help working with "Big Code"
  ☆43Updated 3 years ago
• mast-group / codemining-core
  A set of tools for extracting tokens and ASTs from code
  ☆22Updated 6 years ago
• learnbigcode / learnbigcode.github.io
  Website for Learning from "Big Code"
  ☆29Updated 3 years ago