Hey Meltano community,
As we’ve teased a couple of times over the past few months, we’ve been working on something new that we’re excited to share with you today!
As you may have noticed if you’ve been in our Slack community lately, there’s been a growing amount of interest in Meltano from software teams building products powered by their customers’ data, in addition to our usual audience of data teams building internal data platforms. The need for software engineers to build features on top of data from various sources has been growing due to the rise of AI, and Meltano is a natural fit as our goal has always been to bring software development workflows and best practices to data integration with our open source code-first approach. Turns out that software engineers who need to do data engineering work absolutely love a tool built for data engineers who want to work more like software engineers!
When we launched Meltano Cloud earlier this year, the interest from this part of the community also stood out. From talking to these software teams, it became clear that they have much lower tolerance for self-managing Meltano or any other part of the data stack than data teams. To them, the process of getting and processing their customers’ data is undifferentiated compared to their competitors, and it’s only in using the data that they create value for their customers. It’s not a good use of their software engineers’ time to learn data engineering and build and maintain a data platform, and data engineers are still difficult to hire. They’d rather maintain as little data infrastructure as possible and instead focus completely on building the features that use the data.
Meltano and Meltano Cloud were the best options these software teams found for the Extract & Load stage because of their flexibility and programmability, but they’d rather not have to build and maintain a data platform with 4-to-10 different components at all. There are a lot of products out there that aim to help software teams integrate external data in their products, but these have gone too far in the other direction and have become inflexible black boxes that these teams knew they were going to outgrow, leaving them with only less-than-ideal options.
At the most fundamental level, our team has always been motivated by unlocking the potential value of data, as we believe this will increase productivity, innovation, and progress – both within an organization and in society as a whole. Since Meltano was founded inside GitLab, the best way to do that has been to make data integration available to more people and organizations than ever through open source, and to boost data teams’ productivity, collaboration, and confidence in their output by enabling them to use software development workflows and best practices.
This year, it became clear that the bottleneck in unlocking the potential value of data has shifted from data integration on data teams to data engineering on software teams. So we decided to use our unique experience at the intersection of data and software engineering to do something about it by building on top of Meltano and Meltano Cloud to deliver software teams the data engineering platform that they deserve, tailor-made for building products on top of their customers’ data.
We’ve been working on this with design partners for a few months now, and I couldn’t be more excited to share it with you today: Arch is the bridge between your customers’ data and your code, the data backend for your AI product. I’ve written more about the motivation and long-term vision for Arch in this blog post, and I encourage you to read the product launch blog post by my co-founder Taylor if you’d like to learn more about what Arch can actually do today and what we’ve got in store for the coming months.
While Arch replaces Meltano Cloud as our paid managed offering, you can rest assured that the Meltano project and community that you know and love aren’t going anywhere. Arch is built on top of Meltano and we can’t bring data engineering capabilities to software teams without great underlying data engineering tooling. We will continue to work with you in the Meltano Slack community and GitHub repos every day to make sure it’s the best possible data integration solution for your data platform and our Arch customers. Because we know that not every data engineer using Meltano will want to constantly hear about Arch, and vice versa for every software engineer using Arch, we’ll have separate websites, newsletters, and communities for each. Of course, you’re more than welcome to join both 🙂
We’d love to hear what you think; feel free to reach out in the Meltano Slack community or join us tomorrow for Office Hours!