dspinellis / tokenizerLinks
Convert source code into numerical tokens
☆65Updated 2 years ago
Alternatives and similar repositories for tokenizer
Users that are interested in tokenizer are comparing it to the libraries listed below
Sorting:
- Contains the code for our ICSE 2020 paper: Big Code != Big Vocabulary: Open-Vocabulary Language Models for Source Code and for its earlie…☆84Updated 2 years ago
- A toolkit for pre-processing large source code corpora☆47Updated 2 years ago
- cppminer produces a code2seq compatible datasets from C++ code bases.☆23Updated 5 years ago
- Pyc-cfg is a pure python control flow graph builder for almost all Ansi C programming language.☆54Updated 7 years ago
- ☆26Updated 4 years ago
- Sequence-to-Sequence Learning for End-to-End Program Repair (IEEE TSE 2019). Open-science repo. http://arxiv.org/pdf/1901.01808☆86Updated 2 years ago
- an implementation of "code2vec: Learning Distributed Representations of Code"☆30Updated last year
- srcML Toolkit☆141Updated last week
- Repository for Deep API Learning (DeepAPI)☆55Updated 3 years ago
- Implementation of "Automatic Source Code Summarization with Extended Tree-LSTM"☆36Updated 2 years ago
- Reproduce the results of Tree-based Convolutional Neural Network (TBCNN)☆39Updated 2 years ago
- Official implementation of our work, A Transformer-based Approach for Source Code Summarization [ACL 2020].☆193Updated 3 years ago
- Dataset and code corresponding to Associating Natural Language Comment and Source Code Entities (AAAI 2020)☆20Updated 4 years ago
- ☆67Updated 3 years ago
- mwcvitkovic / Open-Vocabulary-Learning-on-Source-Code-with-a-Graph-Structured-Cache--Code-PreprocessorLibrary for preprocessing java source code into Augmented ASTs, as per the paper Open Vocabulary Learning on Source Code with a Graph-Str…☆21Updated 6 years ago
- TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)☆89Updated 3 years ago
- A tool to convert nodes in an Abstract Syntax Tree into vector embeddings☆75Updated 3 years ago
- ☆24Updated 3 years ago
- Probabilistic API Mining☆53Updated 7 years ago
- A set of tools for extracting tokens and ASTs from code☆22Updated 7 years ago
- Set of tools to help working with "Big Code"☆43Updated 3 years ago
- Website for "A Survey of Machine Learning for Big Code and Naturalness"☆290Updated 6 months ago
- The dataset for the variable-misuse task, used in the ICLR 2020 paper 'Global Relational Models of Source Code' [https://openreview.net/f…☆22Updated 5 years ago
- Neural Variable Renaming for Decompiled Binaries☆44Updated 5 years ago
- Code clone detection; clone-related bug detection; sematic clone analysis☆219Updated last year
- [ICML 2020] DrRepair: Learning to Repair Programs from Error Messages☆195Updated 4 years ago
- Neural Code Comprehension: A Learnable Representation of Code Semantics☆213Updated 9 months ago
- Summarizing Source Code using a Neural Attention Model - CODENN☆239Updated 2 years ago
- ☆45Updated 3 years ago
- Paper Artifacts for "Aroma: Code Recommendation via Structural Code Search"☆59Updated 3 years ago