During working with service based companies I was not aware about all these terms of product development. What are the product development steps, how to develop a good product, what are the tools and technologies to develop a good product, integration, deployment steps, what is system monitoring etc. and all.
System monitoring has an important role. Suppose we have to built a system
How do we get to know if the system is working perfectly or not, right?
So to get to know how the system is working, system monitoring is a really important part and it’s a regular and continuous process 24/7. How much traffic is coming, from where the traffic is coming, from what platforms the most of the traffic is coming and how many exceptions are coming.
Is there any critical error or exception in the system?
If yes then what is the frequency of that error or exception.
How many times load increased on application or database servers?
All these questions are related to our system and force us to think about how we can improve the performance of our system. I learned from working with startups that
“If we can’t major we can’t control if we can major we can control it easily.”
If we are not monitoring and majoring our system on a daily basis and we are going to run a deal then how can we assume that our deal would be successful, right?
Because we don’t know where our system can break during the deal and what kind of errors can occur. After continuous monitoring of a system over a period of time we can improve the performance of our system and can make it stable and scalable.
In startups we hit a trial with multiple products. Suppose if you are working on a product everything is discussed and finalized. What we have to develop then next one hour or next day it might be possible that the idea gets dropped and now you have to work on another product. It’s a normal behavior of startups so don’t worry if you are facing that kind of challenge.
Initially I was frustrated. Many times I thought
What is this?
What are they doing?
It is obvious if you have spent a lot of time understanding the requirements, designing a system flow, architecture and you setup a project for development then suddenly someone says stop and do not work on this project. It’s on hold now you have to work on another project which is most urgent and you have to deliver it in the next 10–15 days.
How will you feel?
But dont worry its a culture of startups and that’s the reason they are called startups. In the initial days I was so frustrated but then slowly slowly I understood the concepts of startups.
Let me share with you an incident that happened with me.
The founders of startups always think about new innovation. I can say that they always think like scientists. If anything new comes in their mind they try to implement that as soon as possible and how soon a POC(Proof of Concept) can be done for that. One day a new thought came to the founder and he shared that with all of us. The next day there was an investor’s visit in our office. You know in startups investor visits are normal because they give the funds to the startups. Most of the startups depend on investors. Few of them hardly think about how to make it profitable to run it for a long time. When investors stop giving funding it’s the end of the startup and now it’s converted into an end up opposite of to startup.
There was a long meeting with investors and all the product’s demos were given to the investors. The new Idea pitched to the investors. They liked it and were happy to invest and they asked to develop a product as soon as possible. They asked us to develop and launch it within 20 days. We were happy and agreed to develop and launch the same.
Within 3 days we designed the system flow, system architecture, database architecture, product wireframes and gave it to the design team. A team of 5 people were prepared for the development. Before the product’s design completion the team completed the backend work in 3–4 days. Now as the design came the backend team started work on its integration of APIs with the frontend team. The team finished its development within 10 days and after code review and code review changes it was pushed into QA. The QA team took 3–4 days in testing and we were ready to go live within 15 days of what we committed to the investors.
But before going live our founder suggested we give a demo first to the investors then we will launch it. We gave a demo to the investors and it was successful. The investors were so happy and appreciated the entire development team. They were impressed with the team work, dedication and commitment of the team. They suggested a few minor changes and asked to go ahead with those changes.
After the product launch our founder decided to run a deal the next day for the product as the marketing team already started the marketing campaigns after the development of the product.
Because the product was developed in such a hurry that we skipped the system monitoring implementation for that. The system monitoring configuration was not a big challenge but we went live with required features only so that we could go live with as per our commitment to the investors.
We were not aware how the system would behave because the system was a newly launched product and the system monitoring part was also missing.
Can you guess what happened during the deal?
The system crashed 10–12 times.
There was a huge load on database servers due to traffic and slow queries.
All the devops people were busy and trying to find out the root causes of what’s going on in the system. Continuously they were checking the requests in the system and based on the failed requests finding the issues. They were busy finding out the slow queries and giving them to the development team. They were caching a few queries and requests. Overall it was a good experience. We learned a lot of things that single day itself and understood the importance of monitoring a system on a regular basis.
I hope you can easily understand what I mean to say overall. The conclusion is that to make a system stable and scalable, monitoring is much required.
Lets understand what system monitoring is?
System monitoring in software development refers to the process of continuously observing and measuring the performance and behavior of a software system or application. The purpose of system monitoring is to detect potential issues or anomalies in the system, so that they can be addressed proactively before they cause significant problems for users.
System monitoring involves the collection and analysis of data related to various aspects of the system, such as resource utilization, network traffic, database queries, and user activity. This data is often collected using specialized software tools called monitoring agents or probes, which run on the system being monitored and collect data at regular intervals.
Once the monitoring data has been collected, it is typically analyzed using a monitoring system or dashboard, which provides real-time and historical views of the system’s performance. This allows developers and system administrators to identify patterns and trends in the system’s behavior, and to take corrective action when necessary.
In summary, system monitoring is an essential part of software development, as it allows developers to proactively detect and address potential issues with their software systems, ensuring that they are reliable, performant, and meet the needs of their users.
Now you would be thinking about the monitoring, right? What are the different system monitoring tools?
Yes, there are various types of system monitoring tools that are used in software development.
Some of the most common ones include:
- Performance Monitoring Tools: These tools help in monitoring the performance of the system, including CPU usage, memory usage, disk usage, network usage, etc.
- Log Monitoring Tools: These tools help in monitoring the logs generated by the system, including application logs, system logs, error logs, etc.
- Alerting and Notification Tools: These tools help in notifying the developers when a particular event or threshold is met, such as when the system is overloaded or when a critical error occurs.
- Tracing Tools: These tools help in tracing the execution of a program, including its function calls, inputs, outputs, and other details.
- Debugging Tools: These tools help in identifying and resolving errors in the code, including syntax errors, logical errors, runtime errors, etc.
- Profiling Tools: These tools help in identifying performance bottlenecks in the code, including slow functions, memory leaks, etc.
- Capacity Planning Tools: These tools help in estimating the resource requirements of the system, including CPU, memory, disk space, etc.
- Security Monitoring Tools: These tools help in monitoring the security of the system, including identifying and preventing security breaches, unauthorized access, etc.
All these system monitoring tools are useful to improve a system performance, make it stable and scalable from time to time. We cannot make a system scalable and bug free the very next day after the deployment. It takes time to make it stable and scalable. After regular monitoring and over the period of a time we can make a system bug free, stable and scalable by using all these tools.
EXERCISE FOR YOU
Implement the required system monitoring tools like Performance Monitoring Tools, Log Monitoring Tools, Alerting and Notification Tools, Capacity Planning Tools and Debugging Tools at least if not all. All these tools are required and necessary tools to monitor a system 360 degree view. If you have already implemented all the required tools it’s great.