Convert source code into numerical tokens
☆67Jul 27, 2023Updated 3 years ago
Alternatives and similar repositories for tokenizer
Users that are interested in tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Smelling smells using Deep Learning☆47Mar 2, 2021Updated 5 years ago
- C# Data Extraction for "Learning to Represent Edits"☆28Nov 3, 2018Updated 7 years ago
- Repository of the paper 'CodeQueries: A Dataset of Semantic Queries over Code' published in ISEC 2024☆13Apr 21, 2024Updated 2 years ago
- ☆20Nov 6, 2019Updated 6 years ago
- Artifacts and other data for "Code Vectors: Understanding Programs Through Embedded Abstraced Symbolic Traces"☆22Jun 5, 2020Updated 6 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- CD4Py: Code De-Duplication for Python☆24Dec 13, 2020Updated 5 years ago
- Improving Code Readability Classification using Convolutional Neural Networks☆11Apr 18, 2018Updated 8 years ago
- ☆15May 27, 2022Updated 4 years ago
- Calculate the score of a repository based on best engineering practices.☆118Sep 27, 2020Updated 6 years ago
- fastText pretrained models for semantic representations of source code in Java, Python, PHP, C, C++ and C#.☆17Nov 11, 2020Updated 5 years ago
- ☆19Apr 15, 2024Updated 2 years ago
- Machine learning models for MLonCode trained using the source{d} stack☆19Oct 30, 2019Updated 6 years ago
- Contains the code for our ICSE 2020 paper: Big Code != Big Vocabulary: Open-Vocabulary Language Models for Source Code and for its earlie…☆84Mar 24, 2023Updated 3 years ago
- DeepCS: Deep Code Search☆285May 26, 2022Updated 4 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Maven plugin to create HTML report to show dependecies in DSM view.☆14Sep 25, 2026Updated 2 weeks ago
- FaCoY Code-to-Code Search Engine☆34Jan 18, 2019Updated 7 years ago
- A Source Code Tokenizer☆13Oct 30, 2024Updated last year
- ☆15Sep 27, 2025Updated last year
- Babelfish Python client☆17Nov 6, 2019Updated 6 years ago
- ☆17Dec 9, 2022Updated 3 years ago
- Assessing Source Code Semantic Similarity with Unsupervised Learning☆39Feb 27, 2018Updated 8 years ago
- TensorFlow code for the neural network presented in the paper: "code2vec: Learning Distributed Representations of Code"☆1,149Sep 20, 2023Updated 3 years ago
- ProgQuery is a system to extract useful syntactic and semantic information from source code programs and store it in a graph database for…☆17Jan 22, 2025Updated last year
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- A Python tool used for parsing exam questionnaires☆17May 7, 2025Updated last year
- A dynamic method for detecting faults in incremental and parallel builds.☆18Jul 27, 2022Updated 4 years ago
- Hexagon processor module for IDA Pro disassembler☆19Oct 11, 2022Updated 4 years ago
- OCaml library to transform an Llvm control flow graph in an SMT formula.☆13Apr 20, 2018Updated 8 years ago
- ☆24Jun 17, 2021Updated 5 years ago
- moneymoney extension for payback accounts☆13Nov 5, 2017Updated 8 years ago
- A benchmark for evaluating embeddings of identifiers in source code.☆23Aug 23, 2021Updated 5 years ago
- Your library for dynamic language modeling☆70Oct 23, 2018Updated 7 years ago
- Fast relational access to openly-available publication data sets☆106Updated this week
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- An empirical study on patch correctness☆15Nov 5, 2022Updated 3 years ago
- AST factorization: transformation AST of Kotlin source code to a vector☆11Oct 17, 2019Updated 6 years ago
- Source code for the Naturalize project☆57Sep 5, 2015Updated 11 years ago
- source{d} datasets ("big code") for source code analysis and machine learning on source code☆349Nov 27, 2019Updated 6 years ago
- A single-header regex-based lexer library for c.☆12Sep 18, 2022Updated 4 years ago
- ☆14May 28, 2024Updated 2 years ago
- Programmers 'C' library of functions for: resizable strings, file and network access, string tokenization, pattern matching, terminal out…☆14Aug 27, 2026Updated last month