dspinellis / tokenizerLinks
Convert source code into numerical tokens
☆65Updated last year
Alternatives and similar repositories for tokenizer
Users that are interested in tokenizer are comparing it to the libraries listed below
Sorting:
- Contains the code for our ICSE 2020 paper: Big Code != Big Vocabulary: Open-Vocabulary Language Models for Source Code and for its earlie…☆84Updated 2 years ago
- Pyc-cfg is a pure python control flow graph builder for almost all Ansi C programming language.☆53Updated 7 years ago
- Sequence-to-Sequence Learning for End-to-End Program Repair (IEEE TSE 2019). Open-science repo. http://arxiv.org/pdf/1901.01808☆85Updated 2 years ago
- A toolkit for pre-processing large source code corpora☆47Updated 2 years ago
- ☆20Updated 8 years ago
- Implementation of "Automatic Source Code Summarization with Extended Tree-LSTM"☆36Updated 2 years ago
- srcML Toolkit☆138Updated last week
- ☆25Updated 4 years ago
- ☆46Updated 3 years ago
- Deep learning code semantic similarity☆64Updated 6 years ago
- the code for three models introduced in DYNAMIC NEURAL PROGRAM EMBEDDINGS FOR PROGRAM REPAIR (ICLR 18)☆32Updated 6 years ago
- A collection of datasets for machine learning for big code☆56Updated 3 years ago
- MLonCode community effort to implement Learning Distributed Representations of Code (https://arxiv.org/pdf/1803.09473.pdf)☆39Updated 6 years ago
- cppminer produces a code2seq compatible datasets from C++ code bases.☆23Updated 5 years ago
- mwcvitkovic / Open-Vocabulary-Learning-on-Source-Code-with-a-Graph-Structured-Cache--Code-PreprocessorLibrary for preprocessing java source code into Augmented ASTs, as per the paper Open Vocabulary Learning on Source Code with a Graph-Str…☆21Updated 6 years ago
- ☆49Updated 2 years ago
- [ICSE 2021] - InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees☆89Updated 3 years ago
- ☆24Updated 3 years ago
- ☆113Updated 2 years ago
- TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)☆88Updated 3 years ago
- Set of tools to help working with "Big Code"☆43Updated 3 years ago
- Data and Code for Reproducing "Global Relational Models of Source Code"☆84Updated 4 years ago
- ☆13Updated last year
- A PyTorch implementation of `code2vec: Learning Distributed Representations of Code` (Alon et al., 2018)☆37Updated 6 years ago
- ☆26Updated 4 years ago
- Contrastive Code Representation Learning: functionality-based JavaScript embeddings through self-supervised learning☆167Updated 3 years ago
- Repository for Deep API Learning (DeepAPI)☆54Updated 3 years ago
- PyTorch's implementation of the code2seq model.☆62Updated 11 months ago
- Neural Variable Renaming for Decompiled Binaries☆44Updated 5 years ago
- ☆58Updated last year