Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• shramos / pyc-cfg
  Pyc-cfg is a pure python control flow graph builder for almost all Ansi C programming language.
  ☆53Updated 7 years ago
• mast-group / OpenVocabCodeNLM
  Contains the code for our ICSE 2020 paper: Big Code != Big Vocabulary: Open-Vocabulary Language Models for Source Code and for its earlie…
  ☆84Updated 2 years ago
• bdqnghi / ast-node-encoding
  A tool to convert nodes in an Abstract Syntax Tree into vector embeddings
  ☆75Updated 3 years ago
• hvdthong / PatchNetTool
  ☆26Updated 4 years ago
• srcML / srcML
  srcML Toolkit
  ☆139Updated this week
• giganticode / codeprep
  A toolkit for pre-processing large source code corpora
  ☆47Updated 2 years ago
• Kolkir / cppminer
  cppminer produces a code2seq compatible datasets from C++ code bases.
  ☆23Updated 5 years ago
• danhper / bigcode-tools
  Set of tools to help working with "Big Code"
  ☆43Updated 3 years ago
• guxd / deepAPI
  Repository for Deep API Learning (DeepAPI)
  ☆54Updated 3 years ago
• ASSERT-KTH / sequencer
  Sequence-to-Sequence Learning for End-to-End Program Repair (IEEE TSE 2019). Open-science repo. http://arxiv.org/pdf/1901.01808
  ☆85Updated 2 years ago
• src-d / code2vec
  MLonCode community effort to implement Learning Distributed Representations of Code (https://arxiv.org/pdf/1803.09473.pdf)
  ☆39Updated 6 years ago
• mast-group / api-mining
  Probabilistic API Mining
  ☆53Updated 7 years ago
• spcl / ncc
  Neural Code Comprehension: A Learnable Representation of Code Semantics
  ☆213Updated 7 months ago
• mwcvitkovic / Open-Vocabulary-Learning-on-Source-Code-with-a-Graph-Structured-Cache--Code-Preprocessor
  Library for preprocessing java source code into Augmented ASTs, as per the paper Open Vocabulary Learning on Source Code with a Graph-Str…
  ☆21Updated 6 years ago
• sh1doy / summarization_tf
  Implementation of "Automatic Source Code Summarization with Extended Tree-LSTM"
  ☆36Updated 2 years ago
• microsoft / dpu-utils
  Utilities used by the Deep Program Understanding team
  ☆102Updated 2 years ago
• bdqnghi / tbcnn.tensorflow
  Reproduce the results of Tree-based Convolutional Neural Network (TBCNN)
  ☆40Updated 2 years ago
• mast-group / codemining-core
  A set of tools for extracting tokens and ASTs from code
  ☆22Updated 7 years ago
• parasol-aser / deepsim
  Deep learning code semantic similarity
  ☆64Updated 6 years ago
• jjhenkel / code-vectors-artifact
  Artifacts and other data for "Code Vectors: Understanding Programs Through Embedded Abstraced Symbolic Traces"
  ☆22Updated 5 years ago
• eth-sri / TFix
  ☆67Updated 3 years ago
• skyhover / Deckard
  Code clone detection; clone-related bug detection; sematic clone analysis
  ☆219Updated last year
• srcML / srcSlice
  Lightweight tool for slicing
  ☆35Updated this week
• ml4code / ml4code.github.io
  Website for "A Survey of Machine Learning for Big Code and Naturalness"
  ☆290Updated 5 months ago
• Mondego / SourcererCC
  Sourcerer's Code Clone project
  ☆213Updated 3 years ago
• micheletufano / NeuralCodeTranslator
  Neural Code Translator provides instructions, datasets, and a deep learning infrastructure (based on seq2seq) that aims at learning code …
  ☆38Updated 6 years ago
• DeepTyper / DeepTyper
  ☆49Updated 5 years ago
• JetBrains-Research / astminer
  A library for mining of path-based representations of code (and more)
  ☆292Updated last year
• soarsmu / AutoPruner
  AutoPruner: Transformer-based Call Graph Pruning (ESEC/FSE 2022, Research Track)
  ☆22Updated last year
• pcyin / dire
  Neural Variable Renaming for Decompiled Binaries
  ☆44Updated 5 years ago