Repository navigation
[C4GT] Performance, Cost Optimization, Benchmarking #28
Description
Project Details
Text2SQL is an application that allows users to interact with their data using natural language queries. Currently, it only supports SQL-based querying but the implementation is not limited to that. Text2SQL provides APIs to generate the appropriate query (SQL or otherwise) and return the data you need.
Features to be implemented
Token Optimization
Improve token usage with OpenAI
Alternate Models Evaluation
Models to be evaluated
- WikiSQL has a some models that can used
- Spider and SparC has some more of these
- RAT is a brilliant implementation of this
- Implement DSP #36
- Evaluate SQL-PALM #37
Domian Mapping to Schema
- Solve for cases when the DB/Tables are not having intuitive names
- Solve for cases where the data in a dataset is needed to figure out viable filters
Test Cases/Benchmarking
Add public test cases to test out the current model.
Learning Path
Complexity
Complex
Skills Required
Python, Knowledge of HuggingFace Transformers, NLP, SQL, Databases.
Name of Mentors:
Project size
8 Weeks
Product Set Up
See the setup here
Acceptance Criteria
- Evaluation Matrix of Model vs Use Case
- Solve for a single Education domain and test if on a new schema
- Run test cases and update benchmarks
- Token usage chart to be shared showing improvements on benchmarks with smaller prompts
C4GT
This issue is nominated for Code for GovTech (C4GT) 2023 edition.
C4GT is India's first annual coding program to create a community that can build and contribute to global Digital Public Goods. If you want to use Open Source GovTech to create impact, then this is the opportunity for you! More about C4GT here: https://codeforgovtech.in/
Activity
[-][C4GT] Performance/Token Optimizations[/-][+][C4GT] Performance, Cost Optimization, Benchmarking[/+]HemanthSai7 commentedon May 18, 2023
Hello @ChakshuGautam ,
I am interested in contributing to this project. Could you please clarify the feature Alternate Model Evaluation? Does it mean trying out the models used in WikiSQL etc and reporting the results?
ChakshuGautam commentedon May 18, 2023
Yes @HemanthSai7. But not with their data - it needs to be a complete cycle of training the model for a domain and seeing the results. Can you start with creating test data first? Also we don't need to do for all, just some promising ones. I am looking at 3 max based on literature review - the ones that have been evolved the most.
HemanthSai7 commentedon May 18, 2023
fibonacci35813 commentedon May 19, 2023
ChakshuGautam commentedon May 20, 2023
Hey guys, let me break this down further and share it by EoD today.
ManasaKaza commentedon May 23, 2023
dixitdeeksha commentedon May 25, 2023
rishabhv471 commentedon May 25, 2023
suyashgautam commentedon May 25, 2023
Hey @rishabhv471 , You can start by setting the project up in a Gitpod environment or in your local. For Gitpod you can follow my video. For local setup you can you can follow the readme. If you face any issues or if you have any question you can ping in the discord channel or you ping me. Will be happy to help. Looking forward to your contribution.
prajak002 commentedon Jun 11, 2023
AmanGadadare commentedon Jun 12, 2023
ChakshuGautam commentedon Jun 20, 2023