Scientific Financial Systems appreciates the power of Snowflake with the scalability, performance, and burstability that it offers our customers. With its recent Snowpark Container Services (SPCS) offering, Snowflake now provides even more value. The allure of SPCS lies not only in its promise of lower latency and enhanced security but also in its capacity for easier scalability and streamlined deployment using SnowSQL as Infrastructure-as-Code (IaC). The team here at SFS is very excited to share that we are migrating Quotient to the Snowpark Container Services model. This migration offers us a tremendous opportunity to leverage advanced database interactions and fortify our partnership with Snowflake.
Understanding the Old Architecture
Quotient previously operated as a containerized application using Docker Compose. This setup allowed us to bundle our application components (Nginx, Redis, Postgres, and Django) into isolated containers, making deployment and management more consistent across different environments. To support this architecture, we developed a suite of Python scripts that served several crucial functions: deployment automation, update automation, CI/CD operations, Django admin automations, data environment population, and other functions.
While this architecture provided a solid foundation and was leaps and bounds above what we started with, it also posed several challenges. The complexity of managing numerous scripts and ensuring their compatibility with our internal environments as well as our client’s environments could be cumbersome. Moreover, as our application and user base grew, we faced limitations in security, scalability, and orchestration capabilities that Docker Compose alone could not address.
Additionally, we sought to enhance our security guarantees for clients. Migrating to SPCS allowed us to leverage advanced security features, such as fine-grained access controls, robust encryption mechanisms, and secure data-sharing capabilities. These features ensure that our clients’ financial data remains protected, complies with industry standards, and mitigates the risk of breaches and unauthorized access. This shift not only bolstered our security posture but also increased client confidence in Quotient’s ability to safeguard their sensitive information.
Finally, tying all of these scripts together alongside an installer that could support many different environments and setups proved to be a significant time and resource burden. This experience highlighted the need for a more robust solution, leading us to explore Snowpark Container Services as a potential upgrade for our infrastructure.
Seeing Further Faster with Snowpark Container Services
The migration to SPCS happened in multiple phases. The initial phase involved migrating our existing support infrastructure including secrets management, image repositories, and the logic for populating Quotient tables on Snowflake. This step was relatively straightforward, thanks to Snowflake’s intuitive SnowSQL commands. We modularized each step into respective SQL files and folders which significantly improved the readability, documentation, and understanding of our infrastructure code. For our custom scripts that populate various Quotient-needed tables on Snowflake, we wrote basic SQL to upload the necessary CSV files onto an encrypted stage. The SQL code then ingested this data into tables based on the CSV headers and column data types.
The next phase involved writing the infrastructure code to deploy our containers, a complex task that necessitated standardizing much of our Django and Nginx code. For instance, we audited all our REST endpoints and Django views to ensure they returned standardized response codes and objects. This ensured our app complied with Snowflake’s proxy service which sits between the public endpoint and the containers. Additionally, we modified our Snowflake connector code to utilize the OAuth flow, which provides us with a faster and more secure connection to Snowflake data.
We also revamped our CI/CD pipelines. Previously, we used GitHub Actions code that SSHed into servers to run various scripts. We replaced this with GitHub Actions code that works with SnowSQL to update and run tests on our containers. This transition not only streamlined our deployment process but also enhanced the efficiency and reliability of our CI/CD operations.
A significant benefit of migrating to SPCS was the ability to separate our Docker Compose application into multiple SPCS services. This segmentation allowed us to scale different parts of our application independently, optimizing resource allocation and reducing operational costs. For example, we can spin down our computational services while leaving the front-end up and accepting requests. As soon as a request comes in, we can then spin back up the cost-intensive computational services. This dynamic scaling capability enables us to manage resources more efficiently, significantly cutting costs while maintaining performance.
Migrating to Snowpark Container Services has transformed our infrastructure, offering enhanced performance, scalability, and security. While the migration process involved several complex steps, the benefits we have reaped make it a worthwhile investment. From modularizing our support infrastructure to standardizing our codebase and implementing robust CI/CD pipelines, the transition to SPCS has positioned Scientific Financial Systems for future growth and success. We are now integrating our Quotient application on Snowpark Container Services with the Snowflake Native App as part of the Private Preview for this combined offering. Stay tuned for an announcement when we publish Quotient to the Snowflake Marketplace!