Client Description
Client X is a leading financial data management solutions vendor for top-tier banks, clearing firms, trading exchanges, and investment managers around the world, providing derivative data across multiple asset classes to feed institutional risk systems.
Project Description
Client X’s legacy solution for reference and market data management was incapable of handling the continually increasing data volumes and evolving functional requirements their customers demanded. Additionally, the prior system came with significant dependencies on cost prohibitive software licenses and a requirement for on-site/local deployment. The client mandate was to completely replace their legacy product with a new back-end and front-end using a modern, and predominantly open source technology stack supporting cloud agnostic deployment.
The new product included major functional enhancements and nonfunctional requirements:
- Storage and retrieval of bi-temporal data
- Exposure of Streaming, REST, and Spark interfaces for data retrieval and analytics
- A redesigned front-end UI and BFF (UI screenshot below)
- Robust nonfunctional requirements for High Availability, Scalability, Performance and Security
Project Scope
Sabal Tech was tasked with bringing in a cross-functional scrum team responsible for all aspects of the SDLC, including a Technical Delivery Manager who interfaced with Client X product managers and senior executive staff, as well as senior stakeholders at Client X’s clients. The Sabal team built and managed internal development and testing environments in both AWS and GCP cloud infrastructures. The use of automation was highly leveraged within the scope of DevOps, cloud infrastructure, and quality assurance.
When the product was ready for UAT and subsequent Production deployment within the infrastructure of Client X’s customers, the Sabal Tech team interfaced directly with their customer to facilitate deployment and transition to the new product. This often required on-site visits to work side by side with the relevant stakeholders.
Scope included:
- Participated in field study with Client X’s sponsor client to gather requirements
- Managed all aspects of the development lifecycle using agile scrum framework
- Established and enforced scrum best practices and set up Atlassian suite of project management tools
- Managed all members of the cross-functional scrum team, including team members employed by Client X
- Responsible for all aspects of technical design and architecture for back-end and front-end
- Responsible for all aspects of back-end development implementation for MVP and numerous subsequent major releases
- Responsible for all aspects of development for front-end MVP
- Implemented a GraphQL API backend for the web front-end
- Exposed product to end user through multiple programmatic interfaces: Spark on YARN, REST API and a language agnostic Streaming API
- Implemented Authentication and Authorization on all exposed interfaces
- Implemented fully automated QA test suites and executed end-to-end testing
- Created full suite of orchestration and configuration management tools in Ansible and Terraform to support deployments in both AWS and GCP environments
- Created full set of AWS CloudFormation templates for AWS deployments
- Utilized Kubernetes and Amazon EKS for container management
- Established fully automated CI development pipeline
- Built and managed scalable development and testing environments in both AWS and GCP infrastructures Duration 30 months of system Architecture and active Development. Technical Advisory, DevOps and support for Production deployments ongoing
Project Phases and Resources
Phase 1:
- Field Study and requirements gathering
- Resource and infrastructure planning
- 1 Technical Delivery Manager
- 2 Back-End Architect
- 1 Full Stack Architect
- 1 SysOps / IT Infrastructure Manager (50%)
Phase 2:
- Back-end system architecture and technical design
- Development project setup
- Establish cloud infrastructure for early stage development environment
- Establish project management tools and scrum framework
- Build product backlog
- 1 Technical Delivery Manager (working with Product Owner from Client X)
- 2 Back-End Architect
- 1 Full Stack Architect (50%)
- 1 SysOps / IT Infrastructure Manager (50%)
- 1 DevOps Engineer (50%)
- 1 QA Automation Engineer (50%)
Phase 3:
- Commence back-end development with ongoing architecture
- Front-end and BFF system architecture and technical design
- Establish QA Automation framework
- Establish automated CI/CD pipeline
- Begin automating cloud orchestration in AWS and GCP
- Optimize cloud infrastructure resource planning
- 1 Technical Delivery Manager
- 2 Back-end Architect
- 3 Senior Back-End Developer
- 1 Full Stack Architect
- 1 QA Automation Engineer
- 1 DevOps Engineer
- 1 SysOps / IT Infrastructure Manager (50%)
Phase 4:
- Ongoing back-end development and architecture (MVP delivered to internal testing at end of phase)
- Commence front-end development with ongoing architecture
- QA test case design and implementation
- Ongoing cloud orchestration in AWS and GCP
- Planning for monitoring and logging solution
- Enhance and optimize CI/CD pipeline
- 1 Technical Delivery Manager
- 2 Back-end Architect
- 3 Senior Back-End Developer
- 1 Full Stack Architect
- 1 Senior Full Stack Developer
- 2 QA Automation Engineer
- 1 DevOps Engineer
- 1 SysOps / IT Infrastructure Manager (50%)
Phase 5:
- Ongoing front-end development and architecture (MVP delivered to internal testing during phase)
- Back-end optimization, refactoring, and defect resolution
- Enhance technical documentation
- Implementation of logging and monitoring solution
- Execute automated cloud orchestration in AWS and GCP
- Execute automated End-to-End testing
- Execute NFR testing
- Enhance and optimize CI/CD pipeline
- 1 Technical Delivery Manager
- 2 Back-end Architect
- 3 Senior Back-End Developer
- 1 Full Stack Architect
- 1 Senior Full Stack Developer
- 2 QA Automation Engineer
- 1 DevOps Engineer
- 1 SysOps / IT Infrastructure Manager (50%)
Phase 6:
- Release back-end and front-end MVP products to UAT
- Refine and enhance product functionality based on feedback
- Engage with Client X’s foundation client for implementation planning
- Manage deployment of solution in Client X’s foundation client’s infrastructure
- Facilitate systems migration for Client X’s foundation client
- Facilitate technical knowledge transfer to Client X team
- 1 Technical Delivery Manager
- 1 Back-end Architect
- 1 Senior Back-End Developer
- 1 Full Stack Architect (50%)
- 1 Senior Full Stack Developer
- 1 QA Automation Engineer
- 1 DevOps Engineer
- 1 SysOps / IT Infrastructure Manager (50%)
Phase 7:
- Ongoing Technical Advisory
- Manage and support deployment in production environments for additional customers of Client X
- 1 Back-end Architect
- 1 DevOps Engineer
- 1 Senior Full Stack Developer
Project Outcome
Client X’s nextgen reference and market data management solution was deemed production ready in Q3 of 2019. In the last 12 to 16 months the product has successfully been implemented, or is in the process of being implemented, at 3 global top-tier banks and 1 major international financial exchange. The size of the Sabal Tech team has been scaled back to coincide with the product/project maturity, but key members remain active in a development and devops capacity.
Why the project was a good fit for Sabal Tech
- Allowed Sabal Tech to leverage their significant domain knowledge in financial markets and reference data
- Provided the opportunity to work with a modern technology stack
- The team was enticed by the challenge of managing very large data volumes while dealing with the complexity of handling bi-temporal data
- The team was attracted to the challenge of implementing multiple interfaces which could support timely queries across the entire bi-temporal data set
- Provided the opportunity to build a cloud agnostic product and execute deployments in both AWS and GCP
- Automation was highly encouraged by Client X, which aligns with Sabal Tech’s philosophy
