An example on what bad AI agencies do when they say:
"start with one workflow and automate it with AI!"
- Company: Food distribution (2,000 employees)
- Workflow: R&D for Trends, Pricing, and Competitive Analysis
- Cost: $50k
- Problem: 150 employees across separate teams spend roughly 2-4 hours a day checking websites for trends, pricing, and competitor data.
They upload their findings to the CRM and prepare pitch decks for management to show clients. The end goal is to use that research to sell their own inventory.
AI agency runs a full audit and sees this is a good workflow to automate.
They built web scrapers that allow for employees to visit websites and click on items they want to add to their "list". The scraper would then pull prices, images, and updates on those items automatically and send the results to the CRM.
They finished it up. $50k for the build, $5k a month for maintenance. The agency had a dinner over it talking about how they're so innovative and the agency is impressed with how much time they're saving the company, while the CTO and CEO were just nodding and smiling.
The AI agency had set the build scope to only 10 main websites, but really there were over 50 websites that employees went through. The employees hated having to constantly ask for new websites to be supported, only to find out the AI agency's monthly retainer of $5k only covered maintenance and not new development.
Employees were frustrated because they said they spent most of their time finding out what was available in the market and what to look for. They were confused to why the agency would build a web scraper at the "easiest" part.
The data itself was the biggest problem. Every new scrape overwrote the previous data in the CRM. If an employee corrected a result, the next scrape would overwrite it. No history log of what changed or who changed it.
When I asked why they chose to not use a data warehouse, they corrected me and said that they did.
It was a poorly setup Supabase instance. Great tool, terrible setup.
Then they said that it was all accurate and that the employees were the ones overwriting data that was wrong. A quick audit proved otherwise.
After talking to the sales team, who relies on the R&D data, they said getting the R&D done faster is second to the quality of the data and the signal they get from it. They wanted forecasting, past history, and analytics based on what worked and what didn't work.
Now take a step back. The workflow was supposed to support trends, pricing, and competitive analysis... how were they supposed to track all of that is the scrape erased the previous data?
Before building any of this, check what data the company already has. There were 100s of spreadsheets. After that, check what you're missing. Do the websites offer APIs? Is there an existing industry tool or data provider worth paying for? What would it cost to build and maintain the scrapers yourself? A quick POC shows that you CAN build a ton of web scrapers but SHOULD you?
The sad part is the the AI agency did a full "AI audit" where the found the spreadsheets and had access to a lot of data, but they still chose to easy way out by building a tool as opposed to truly understanding the business operations and their needs.
Figure out what history is usable, what data is missing, and build the collection tools around that.
If scraping makes sense, build it around the data you need and how often it changes. Keep the history. Track failures. Make sure employees aren't spending their day fixing the same bad results.
Literally do anything the relies on solid fundamentals instead of doing what the agency did by just throwing meeting transcripts into claude to determine the build path.