Before startups I was also not aware about all these terms of system or product development. I worked around 5.5 years in service based companies. But there I didn’t get a chance to learn about all these things which is the major drawback of working with service based companies or well established organizations. People work on small small things days or months to deliver. While in startups you can deliver multiple things within a month and that’s the reason people grow faster working with startups. It’s obvious there would be work load and sometimes pressure but you learn things faster which people don’t learn in 15 or 20 years in other well established organizations.
I worked around 5–6 years as a senior software engineer. I struggled for 8 years to become a technical lead. But after joining a startup I was promoted as technical lead within 3–4 months. The beauty of startups is you have to handle each and everything. So, I started handling each and everything end-to-end.
When responsibilities are given to you, you have to think about that always. I started directly working with product managers. But in the initial days I directly worked with the CEO without any intermediary as a product manager. I designed and developed three major products as ACL(Access Control List), CANP(Communication Alerting and Notification Platform) and State Machine(Handling All The Statuses). I got awarded with a top gear award and promoted as technical lead for the launching of my first product ACL.
The system flow design is not a technical thing. This could be designed by any other person who wants to build a system. We can say that person would be the real product owner of that product. He could be CEO of the company, any business person or a product manager. Generally if you are working with a team of product managers then this could be done by the product manager. But while I was working with product managers, I was always responsible for designing system flow and its architecture.
But the system architecture design is a technical task and this could not be done by a non-technical person. A person who is technical and has a good understanding of technologies and databases is the right person to design a system architecture. Technical architects, senior engineering managers or engineering managers are the right people to design a system architecture. Who have a good understanding about the technologies and system flow. I started designing the system flow and system architectures after joining my second startup while I joined as a senior software engineer there and continued that till I was promoted as senior director of engineering including other responsibilities as well.
Now lets understand what is a concept of system design.
System design is the process of defining the architecture, components, modules, interfaces, and data for a system to satisfy specified requirements. It is a multi-disciplinary approach that combines elements of software engineering, hardware engineering, and user experience design to create a comprehensive plan for building a complex system.
The main objective of system design is to develop a blueprint or roadmap for constructing a system that meets the needs of its users, is efficient, reliable, and can be maintained and scaled over time. System design involves analyzing requirements, identifying potential design solutions, evaluating the trade-offs between different options, and selecting the best design approach.
Some key elements of system design include:
- Requirements Analysis: This involves identifying the needs of the users and stakeholders, and defining the system’s functional and non-functional requirements.
- Architecture Design: This involves defining the overall structure of the system, including the high-level components, modules, and interfaces.
- Detailed Design: This involves specifying the details of the system’s components, such as algorithms, data structures, and control logic.
- Implementation: This involves coding the system components and integrating them into a working system.
- POC(Proof of Concept) and Verification: This involves validating that the system meets the requirements and is free of defects.
- Maintenance and Evolution: This involves maintaining the system over time and making changes to it as needed to keep it up-to-date and meet changes according to user needs or requirements.
There are different types of system design, and the type of system design depends on the context, the complexity, and the goals of the system. Here are some common types of system design:
- Conceptual Design: This type of system design involves identifying the high-level requirements and goals of the system and defining the overall architecture and approach for meeting these requirements.
- Logical Design: This type of system design involves defining the functional components of the system and their relationships. It focuses on the logical flow of data and information in the system.
- Physical Design: This type of system design involves defining the physical components of the system, such as hardware, software, and networking components, and how they are interconnected.
- User Interface Design: This type of system design focuses on the user interface and how users interact with the system. It involves designing the layout, navigation, and visual elements of the user interface.
- Database Design: This type of system design involves defining the data structure and schema for the system’s database. It includes designing the tables, fields, and relationships between them.
- Network Design: This type of system design involves designing the network infrastructure of the system, including the topology, protocols, and security measures.
- Security Design: This type of system design involves designing the security features of the system, such as access control, encryption, and authentication.
But it all depends on the type of system design and on the specific requirements and goals of the system being designed.
Majorly we can divide the entire system design concept into two parts as
- Low Level System Design which is also called as LLD in its short form
- High Level System Design which is also called as HLD in its short form
- Low Level Design(LLD):
Low-level system design is the process of specifying the detailed implementation of a system’s components or modules. It involves converting the high-level system architecture and requirements into a detailed design that can be used for implementation. This level of design focuses on the technical details of how the system will be built and implemented, including algorithms, data structures, and control logic.
In low-level system design, each component or module is analyzed in detail, and decisions are made about how it will be implemented, including the programming language, libraries, and frameworks that will be used. The design includes specifications for interfaces between components, data formats, and storage mechanisms. The design may also include details about performance, such as expected response times and memory requirements.
Low-level system design typically follows the completion of the high-level system design and is a crucial step in the overall system development process. It provides the detailed specifications needed for the actual implementation of the system and ensures that the system is efficient, reliable, and maintainable. A well-designed low-level system design can help to reduce the likelihood of errors or bugs in the final system and can also make it easier to update and maintain the system over time.
We can break down a Low Level Design into two parts as
Designing a Database Architecture
- Selection of database type according to the use case
- Creating required tables for the system
- Identifying the relationships between the tables and designing an ER-Diagram
- Identifying the required indices on particular columns
- Thinking about the optimized way to store the data
- Writing a migration for the database
Designing a System Skeleton
- selection of a skeleton or a framework for project
- defining a structure of classes and models
- Thinking and managing of views
- Designing a UI and UX
- High Level Design(HLD):
High-level system design is the process of defining the overall architecture and structure of a system. It involves identifying the major components and modules of the system, their interactions, and the relationships between them. The main goal of high-level system design is to provide a blueprint or plan for the development of the system that can guide the detailed design and implementation of the system.
In high-level system design, the emphasis is on the functional requirements of the system and how they will be achieved. The design may include diagrams, charts, or other visual representations to help communicate the overall structure of the system. The design may also include specifications for interfaces between components, data formats, and storage mechanisms.
High-level system design typically follows the completion of the requirements analysis and is a critical step in the system development process. It provides a foundation for the detailed design and implementation of the system and ensures that the system is designed to meet the needs of its users. A well-designed high-level system design can help to ensure that the system is efficient, reliable, and easy to maintain and evolve over time.
We can break down a High Level Design into two parts as
Functional Requirements
- Designing a system flow how system will work
- Dividing the system into different modules as functional requirements
Non-Functional Requirements
- Scalability
- Reliability
- Availability
- Performance
- Security
We understood about the basic concepts of system design theoretically now lets understand the system design concepts with help of an example practically as well.
Suppose we have to design a system for an ecommerce platform just like Amazon but not too many features and functionalities only for basic understanding purposes. To design a system first we have to think about how many features we have to keep in our system to start with.
Below are the list of features or we can say functional requirements for our system which we are going to build
- User can register and login
- A functionality to add products or catalog from back-end
- User can able to search or filter their desired products
- End-end add to cart to thank you page a shopping experience
- Different communications like email, sms or push notifications wherever its required
- User can able to track their order in his account
- Return and refund handling
- A CMS management platform for static pages
- A CANP(Communication Alerting and Notification Platform)
First of all we have to design a system flow for these requirements so let’s design a system flow how our system will work. We can easily understand the system flow by the help of the following diagram as
This is a basic system flow diagram for our e-commerce platform. Now based on our requirements and to keep in mind the above system flow diagram we have to design a system architecture.
The following could be a one of system architecture for our system as
A system architecture design is a very complicated process which needs continuous improvements and refinements. I am not saying that this is a perfect system design for an ecommerce system. But for your understanding purpose this could be one of a basic system design approach. Even in the first version or if there is not much traffic then there is no need for a load balancer and multiple servers. In the initial days we didn’t need elastic, redis or mongodb databases. We can go only with the mysql database.
Still there is a room of improvement in this system design as we can use any message broker technique like RabbitMQ or Kafka to improve user experience. But I am keeping a space for improvement and this is an exercise for you.