Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• giganticode / codeprep
  A toolkit for pre-processing large source code corpora
  ☆47Updated 3 years ago
• mast-group / OpenVocabCodeNLM
  Contains the code for our ICSE 2020 paper: Big Code != Big Vocabulary: Open-Vocabulary Language Models for Source Code and for its earlie…
  ☆84Updated 2 years ago
• shramos / pyc-cfg
  Pyc-cfg is a pure python control flow graph builder for almost all Ansi C programming language.
  ☆56Updated 7 years ago
• src-d / code2vec
  MLonCode community effort to implement Learning Distributed Representations of Code (https://arxiv.org/pdf/1803.09473.pdf)
  ☆39Updated 7 years ago
• DeepTyper / DeepTyper
  ☆50Updated 5 years ago
• Kolkir / cppminer
  cppminer produces a code2seq compatible datasets from C++ code bases.
  ☆23Updated 5 years ago
• hvdthong / PatchNetTool
  ☆26Updated 4 years ago
• bdqnghi / ast-node-encoding
  A tool to convert nodes in an Abstract Syntax Tree into vector embeddings
  ☆76Updated 3 years ago
• mast-group / api-mining
  Probabilistic API Mining
  ☆53Updated 7 years ago
• eth-sri / TFix
  ☆69Updated 3 years ago
• srcML / srcML
  srcML Toolkit
  ☆148Updated last week
• jjhenkel / code-vectors-artifact
  Artifacts and other data for "Code Vectors: Understanding Programs Through Embedded Abstraced Symbolic Traces"
  ☆22Updated 5 years ago
• ASSERT-KTH / sequencer
  Sequence-to-Sequence Learning for End-to-End Program Repair (IEEE TSE 2019). Open-science repo. http://arxiv.org/pdf/1901.01808
  ☆86Updated 2 years ago
• sriniiyer / codenn
  Summarizing Source Code using a Neural Attention Model - CODENN
  ☆239Updated 2 years ago
• pcyin / dire
  Neural Variable Renaming for Decompiled Binaries
  ☆44Updated 5 years ago
• tech-srl / PigeonJS
  Demonstration of the path-extraction process shown in the paper "A General Path-Based Representation for Predicting Program Properties"
  ☆24Updated 4 years ago
• sh1doy / summarization_tf
  Implementation of "Automatic Source Code Summarization with Extended Tree-LSTM"
  ☆36Updated 2 years ago
• squaresLab / genprog-code
  GenProg: heuristic, GP-based automatic program repair for C.
  ☆99Updated 4 years ago
• bdqnghi / graph-ast
  A tool to provide the graph representation of the source code based on the paper: "Learning to Represent Program with Graph, ICLR'18"
  ☆50Updated 5 years ago
• spcl / ncc
  Neural Code Comprehension: A Learnable Representation of Code Semantics
  ☆215Updated 11 months ago
• sonoisa / code2vec
  an implementation of "code2vec: Learning Distributed Representations of Code"
  ☆30Updated last year
• guxd / deepAPI
  Repository for Deep API Learning (DeepAPI)
  ☆56Updated 3 years ago
• typilus / typilus
  Code for "Typilus: Neural Type Hints" PLDI 2020
  ☆62Updated 2 years ago
• SpoonLabs / coming
  A tool for mining commits from Git repositories and diffs to automatically extract code change pattern instances and features with ast a…
  ☆98Updated last year
• panthap2 / AssociatingNLCommentCodeEntities
  Dataset and code corresponding to Associating Natural Language Comment and Source Code Entities (AAAI 2020)
  ☆20Updated 5 years ago
• tushartushar / DeepLearningSmells
  Smelling smells using Deep Learning
  ☆47Updated 4 years ago
• danhper / bigcode-tools
  Set of tools to help working with "Big Code"
  ☆43Updated 3 years ago
• mast-group / codemining-core
  A set of tools for extracting tokens and ASTs from code
  ☆22Updated 7 years ago
• ASSERT-KTH / CodRep
  58069 Java source code diffs. http://arxiv.org/pdf/1807.03200
  ☆94Updated 6 years ago
• JetBrains-Research / code2seq
  PyTorch's implementation of the code2seq model.
  ☆62Updated last year