A Python data analysis library that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data easy. Together with NumPy, it's one of the go-to tools for data analysis.. and unless you're truly an expert, it's arguably the package you'll run into even more often than NumPy.
Built on top of that package, pandas-ai is an open-source AI agent for data analysis that applies an LLM on top of pandas.
1. Pandas AI | PandasAI meets a local LLM (feat. ollama)
? 2. Pandas AI | PandasAI with Agent, OpenAI, MySQL
Last time I used a local LLM to turn a dataframe into a SmartDataframe and tried out various things. This time, I'll go through uploading a file directly to a pure? LLM Agent, using the OpenAI API instead of a local LLM, and connecting directly to a MySQL database.
0. References
An open-source AI agent for data analysis
PandasAI - Conversational Data Analysis
PandasAI is a Python library that integrates generative artificial intelligence capabilities into pandas, making dataframes conversational
pandas-ai.com
GitHub - Sinaptik-AI/pandas-ai: Chat with your database (SQL, CSV, pandas, polars, mongodb, noSQL, etc). PandasAI makes data ana
Chat with your database (SQL, CSV, pandas, polars, mongodb, noSQL, etc). PandasAI makes data analysis conversational using LLMs (GPT 3.5 / 4, Anthropic, VertexAI) and RAG. - Sinaptik-AI/pandas-ai
github.com
LLM Agent
1. First, like last time, initialize with the local LLM and build the screen with Streamlit. This time, unlike before, let's build a sidebar-style UI.
from pandasai.llm.local_llm import LocalLLM
import streamlit as st
model = LocalLLM(
api_base="<http://localhost:11434/v1>",
model="llama3"
)
#st.UI-타이틀
st.title("Data analysis with PandasAl Agent")
#st.UI-데이터 세트로드
uploaded_file = st.sidebar.file_uploader(
"upload a CSV file",
type=['csv']
)
2. Then, like last time, when the file is uploaded successfully, we read it and render the first 3 rows as a table. The difference is that we now create an instance of the LLM Agent, hook it up to the model, and have answers come back from the agent rather than from the dataframe.
...
#st.UI-read & write
if uploaded_file is not None:
data = pd.read_csv(uploaded_file)
st.write(data.head(3))
agent = Agent(data, config={"llm":model})
prompt = st.text_area("Enter your prompt:")
if st.button("Generate"):
if prompt:
with st.spinner("Generating response..."):
st.write(agent.chat(prompt))
After uploading the file, here's the result of asking it to 'return the last 3 columns of the dataset'.
And this time, after uploading a slightly larger file, here's the process of asking about missing values and printing out their locations and the corresponding rows.
And here's the result of asking it to draw a boxplot of age broken down by membership type.
OpenAI API
The biggest difference is that instead of from pandasai.llm.local_llm import LocalLLM, we use from pandasai.llm.openai import OpenAI. And there's an additional part for entering the API key.
from pandasai.llm.openai import OpenAI
import streamlit as st
from pandasai import SmartDataframe
import pandas as pd
from pandasai import Agent
OPENAI_API_KEY = "sk-********************************"
llm = OpenAI(api_token=OPENAI_API_KEY)
#st.UI-타이틀
st.title("Data analysis with PandasAl Agent")
#st.UI-sidebar 데이터 세트로드
uploaded_file = st.sidebar.file_uploader(
"upload a CSV file",
type=['csv']
)
#st.UI-read & write
if uploaded_file is not None:
data = pd.read_csv(uploaded_file)
st.write(data.head(3))
agent = Agent(data, config={"llm":llm})
prompt = st.text_area("Enter your prompt:")
if st.button("Generate"):
if prompt:
with st.spinner("Generating response..."):
st.write(agent.chat(prompt))
From this we could see that Age has a strong positive correlation with Total Spend (0.96) and Items Purchased (0.74).
MySQL
First, refer to the link below to install MySQL.
(Mac) MySQL — full uninstall, install, and configuration guide
While putting together a service with a Local LLM Agent, I found myself reinstalling MySQL after a long time. With the policy changes from 8.x onward, the version on my machine being too old, and my memory being fuzzy, I went back to what I'd written before
normalstory.tistory.com
Download a suitable sample dataset (CSV) from Kaggle and import it into a database table.
E-commerce Customer Behavior Dataset
Exploring Customer Engagement and Purchasing Patterns in an E-commerce
www.kaggle.com
from pandasai.llm.local_llm import LocalLLM
import streamlit as st
from pandasai.connectors import MySQLConnector
from pandasai import SmartDataframe
model = LocalLLM(
api_base="<http://localhost:11434/v1>",
model="llama3"
)
my_connector = MySQLConnector(
config={
"host": "localhost",
"port":3306,
"database":"EComUsr",
"username":"root",
"password":"****!",
"table": "customer"
}
)
df_connector = SmartDataframe(my_connector, config={"llm":model})
#st.UI-타이틀
st.title("MySQL with Llama3")
prompt = st.text_input("Enter your prompt:")
if st.button("Generate"):
if prompt :
with st.spinner("Generate response.."):
st.write(df_connector.chat(prompt))
