In 2019, a medical device company approached Augusto for our consulting help on an ecommerce project…
since we were already helping them with their website. Two of the client’s non-technical marketing personnel had been asked by their internal IT team to build functional specifications on a highly technical level, in order to begin the work of moving and transforming data from Salesforce into Shopify.
As we engaged with this client, a couple problems came into focus. First, the internal IT team wanted to fully define a system of disparate and complex entities before they began building the infrastructure to connect them.
Second, the deadline to launch the first Shopify store was quickly approaching. While the team initially had a year to launch the store, six months had passed without agreeing on a plan and way forward.
These problems presented the perfect project for agile development. In this situation, we knew the end goal was to identify products and customers in Shopify that reflected some subset of data in Salesforce and turn orders into opportunities. But we didn’t know how all of the different data points would interact, which ended up being the major roadblock.
Perfection is the Enemy of Progress
To solve the first problem, we had to define the project in such a way that we could make progress without knowing all of the details. Team members from Augusto, the client company, and Praella—a Shopify company that would handle the store implementation—met in Chicago for a two-day workshop.
The goal was to gather enough information to begin work, knowing that we would not answer all of the details but could agree on a way forward.
Over the course of those two days, we identified the basic building blocks in Shopify, mapped them to data in Salesforce, determined items we knew for sure, and outlined a rough idea of the details that we did not know.
Sprinting Forward
To solve the second problem, we knew that it was time to get to work. While we didn’t know all of the answers, we had identified a solution that would assist both the marketing team and the technical team in understanding the data flowing between the two systems.
Augusto began by creating a general functional specification that outlined what needed to be done, the details that we already knew, and the items we still needed to figure out. We then broke that functional specification down into phases:
- Play with Salesforce API to determine best way to retrieve and update data
- Play with Shopify API to determine best way to retrieve and update data
- Create a simple web platform to visualize the data flowing between the two systems
- Begin building up products from Salesforce Product and Pricebook data
- Create and update products in Shopify from Salesforce data
- Begin building up customers from Salesforce Contact and Account data
- Create and update customers in Shopify from Salesforce data
- Build a background job system to ensure data is always flowing and not dependent on the website
- Improve website platform to better show success and errors. Identify the source data and the target data as JSON that can be explored for a better understanding of what is moving between the systems.
The secret sauce was surfacing what was going on in the background. All jobs that moved data in any direction were logged and visible to the marketing team. This was crucial to identifying what moved when, what did not move or errored out, and then allow those errors to influence the data and details that we didn’t yet know.
As we moved into each of those phases, we had a rough idea of what would work, a solid understanding of what it needed to end up as, and iterated through code to figure out what hurdles existed. Those hurdles enabled us to go back to our client with a clear problem and allowed them to ask the correct questions to business owners. Together we cleared the hurdle in a way that we could never have predicted before digging down into the process.
This Agile process can often be counterintuitive. People often wonder: what if you go too far before you realize a solution won’t work? While it’s possible that a wrong decision or assumption can lead to having to scrap some code and rethink it, imagine how much worse off that would be after a year of planning to find out that any or all of your assumptions are wrong. It is better to try and fail early, correcting for those mistakes that you know are coming.
Launching on Time
Work moved along quickly and we were in a good place to launch the ecommerce site on time. But with a week to go before launch, we learned of a new set of data that needed to be migrated to a different system in order to facilitate Single Sign On (SSO).
The system we had built handled product, customer, and order migrations. While these were tied to Salesforce and Shopify, the groundwork was in place to add a new migration type from Salesforce to Azure Active Directory. The only new piece was how to talk to Azure.
Within the week, we were able to add a new migration type, display the flow of data between Salesforce and Azure Active Directory, and determine how to send that Salesforce data to Azure. All of the work we had put into the previous four months enabled us to quickly adapt to a new requirement.
The site launched on time to much fanfare and success. The team was excited and happy with the results, but it was just the start of the project. Over the course of the next year, we updated and expanded the website and the migration code.
- We added diagnostic tools to help fix Customers and Orders if they encountered an error. In most cases, this was due to data that we had not encountered in our dev environments.
- More stores launched in different regions of the world, forcing the website and the code to handle not only a Product, Customer, and Order migrations from Salesforce to Shopify, but to do that with an increasing number of Shopify stores, broken down by global regions.
- We added another migration type from Salesforce to a separate sales tool.
- We made performance improvements to handle the increasingly large amount of migrations that were happening.
- We took on more of the client’s APIs to create digital products as orders from Shopify came in.
While the end goal of what we wanted to build did not change, the details on how we got there, the number and type of migrations, and the features that ended up being built were not something that we could have fully anticipated in 2019. However, what we did build fully earned the name The Integrator that was bestowed upon it.
ETL Happens
The basic premise of this project is nothing new. This was an Extract, Transform, Load project (ETL), and there are automated tools that assist in connecting many major systems. In this case, no automated system would have handled this process out of the box, and that was something we identified early on.
- There were too many identified unknowns to fully vet an automated system.
- There were multiple instances of taking Salesforce Data from more than one location, and adding it to Products and Customers in more than one place in Shopify → basic data, metadata, and tags were all used heavily.
- The marketing team wanted more visibility into what was happening, and needed a way to fix issues without going through the IT team’s support system.
- While not known at the outset, we ended up connecting two more systems with Salesforce using the same method we built for Shopify. For one of those systems, a custom solution was the only option. With the basic premise in place already, it was a relatively trivial process to connect the new system.
Let’s Start a Conversation
Contact Augusto today to start a conversation about custom app development, cloud computing services, software consultation or outsourced software teams.