Click above“ Programmer Xiaohui”, Choice“ Top public address”
Interesting and meaningful articles arrive at the first time!
This article is reproduced from the public address. Code farmer turn over
The war of development and operation and maintenance
Five days ago, Zhang Dafeng's development team delivered a batch of new codes to the operation and maintenance department, This is an important upgrade that users have been looking forward to for a long time, The deployment went very well, Everyone is very happy.
But today's production environmentCPU Continuous approach100%, There are several serversdown Machine. Operation and maintenance boss is furious：“ It's the third time! Zhang Da fat, What happened to your development team?
New code on lineCPU Just100%!”
Zhang Dafang is not willing to show weakness：“ We tested it very well in the test environment, User pressure is much greater than production environment, Code is rock solid, You must have configured something wrong!”
“ Impossible, We are deployed in strict accordance with the steps you require, It must be your code!”
“ Then why is the test environment OK?”
Two executives have been arguing for a long time, There is no good solution, We stayed up all night, Roll back the code to the previous version, Burn incense and worship Buddha, Hope no more problems.
After this toss, The bonus at the end of this year is expected to be up in the air!
Zhang Dafang feels extremely depressed, Difficult mood, Black face came to the tea room and poured a cup of coffee, Sitting there and drinking it again, I feel that the operation and maintenance department is too difficult to cooperate!
Look at the new recruits, No business at all, They want to“ Evasion of responsibility”, Often say：“ I don't know the business, Content of this launch, You should write every step clearly, I just do it, Don't ask why, It's not my responsibility that something goes wrong.” You say it's annoying!
Does he want to be a robot?
Also, Their operation and maintenance team focuses on different people, Someone is responsible for database script execution, Someone is in charge.Web The server, Someone is in charge.SSO ,
Oh my God, I have to pull a bunch of people over for several meetings before I go online, Familiarize them with the operation steps.
This man deployed once, I'm not familiar with it, Next time I change, It's also called human resource pool, Can improve efficiency, But new people need to learn from the leader, How could it not be wrong?! alas......
Memories of Zhang Dafang
After two cups of coffee, Zhang Dafang calms down a little, Think carefully, The reason is that the software itself is too complex, Not only the development is complex, Deployment is also super complex, Every time you deploy, you peel people off.
Zhang can't help but think of the software development and deployment process he has experienced in the past few years.
University time, Do some small projects with the teacher, Development, The tests are all done by one person, Deployment to users is also self-made, Almost no trouble.
Enter a small company after graduation, What is done isC/S Architecture system, With a development team, Test team, The development team wrote the code, Test by test team, After that, you can deploy to the customer.
Generally speaking, the implementers are also the brothers of development or testing, I also worked part-time, Take the deployment document and CD, Go to the customer and install the system on the customer's machine in strict accordance with the steps, Basically, there's no big problem, Even if there's a problem, It can be solved after debugging on site, Big deal, call the development brothers to stay up late together.
Then the Internet wave came, I also switched to this Internet company, Make a car appointment system on the Internet, Provide car appointment service to users, You don't have to go to the customer to install the software, The company operates independently, It's all right to maintain this system.
But the online car Hailing system is comparable to the original stand-alone software,C/S Software is much more complex, Especially in the face of massive users' high concurrent access, All kinds of technical problems need to be solved, The challenge is enormous..
The system is not only complex, Also need to24*7 How to run, Part time developers or testers are no longer able to maintain.
The company has set up operation and maintenance（Operations） department, Responsible for system deployment, Routine maintenance, Monitor. Unprecedented improvement in the status of operation and maintenance personnel, Of course, The demand for their skills has also increased unprecedentedly.
Zhang Dafang has seen an email about the operation and maintenance for recruitment:
Skilled useLinux, unix, windows operating system;
Proficient in all kinds of common server software（tomcat, apache, nginx,redis,mysql...) Configuration and optimization of
Understand the principles of load balancing and high availability, asLVS,Keepalived etc.
Familiar with network principle,TCP/IP Agreement, Master at least one scripting language.
Can use various configuration management and deployment management tools.
in short, Operation and maintenance is almost as important as development.
The gap between development and operation and maintenance
To speed up delivery, Two years ago, Led the development team to implement agile transformation, Successfully transformed the original waterfall development mode into a trot, Agile mode of frequent delivery.
（ picture source：http://www.agilenutshell.com/scrum）
Through agile software development, Put people in need, Developer, The testers pulled together, Forming so-called“ Feature team”, Separate requirements into independent ones, Stories of value to users, Prioritize and develop later, test, It can even deliver several independent requirements every two weeks.
（ Manon turn note, See also《 Vernacular agile software development
The successful agile transformation is highly recognized by the company, They also exported a lot of training.
Although it can be delivered frequently, But they can't go online frequently, The reason is simple： The operation and maintenance guys always want the system to be stable, Stable, Re stabilization, Stability prevails.
So they don't want to go online frequently, Isn't that making trouble for yourself?
It's the opposite of agile software development, Agility is about embracing change !
( Development requirements change, Stable operation and maintenance requirements, Image creation from http://dev2ops.org, Lao Liu redraws)
I figured out the essential contradiction, Zhang dapang knew that he couldn't solve the problem, It must be solved by the upper level.
Zhang Dafang goes to find it immediatelyCTO Bill, I hope he can come up with some good ideas.
Dev + Operations = DevOps
What he didn't expect was, Lao Li, operation and Maintenance Director, is already inBill In the Office, Zhang dafangxin says no, Maybe the villain told me first.
Bill The sight of chubby Cheung, Let him sit down first, Listen to Lao Li“ contradiction” and“ Warfare” Finish.
Lao Li is nagging, It's similar to what I think.
Bill Say with a smile：“ Big fat, The development process of software is basically the requirement-> Development-> test-> deploy, Now your team has put the needs, Development, Test to‘ Unite’ Come together,
It seems you want to‘ Unite’ A new partner!”
“ Is it Lao Li's operation and maintenance department?”
“ You're right.”
“ That's impossible. Our goals are totally different, One needs to change, One needs stability, How can we play together?” Big fat, very surprised.
“ No, In the future, we will focus on business objectives, Take the value of users as the only criterion, The assessment and evaluation mechanism of the team should also be changed, The success of both individuals and teams depends on the overall development- Evaluation within the operation and maintenance life cycle, Development completed a lot of user requirements are not necessarily successful, Operation and maintenance guarantee systemdown It doesn't have to be a success! Only the functions that users want are implemented in time, Deployed successfully, It's only successful if it's used stably.
” CTO Always good at usingMBA To talk.
“ That is to say, we are required to cooperate closely in operation and development?” Lao Li then asked.
“ Yeah. Now there is a hot word calledDevOps, It is to break through the fence between the agile development department and the operation and maintenance department, Form closed loop.”
“ Are we going to add another department, callDevOps department? recruitDevOps Engineer?”
“ No no, If we add another department like this, Didn't another wall be added?
We were supposed to break the gap between development and operation and maintenance teams. In fact, the goal of the operation and maintenance department is not only to make our online car Hailing system more stable and efficient, We need to support the rapid evolution of business—— This is consistent with your goal of agile software development.”
" But we can't deploy it frequently? The contradiction between speed and stability can't be solved." Old Li sighed.
" I know that Zhang's team is implementing the transformation of microservice, In the future, it's not a big Mac application, It is based on micro services, That would be much easier, Frequent deployment is possible, And it's much easier to roll back when something goes wrong, I don't need you to stay up late!"
Zhang Dafang and Lao Li both nodded in agreement.
“ What should I do?”
“ The first is the change of concept, You can't shirk responsibility for each other in the future, Recriminations, And we have to share the responsibility! Development of a project, deploy, Maintain, It's about both of you, Both parties are responsible for the business, What's wrong, You need to work together, Joint solution. You need to give more members when you go back` Brainwashing`.”
Zhang Dafang thought, We just worked together to roll back the code.
“ Also,”,Bill Take a look at Lao Li,
“ Operation and maintenance personnel should also understand the business,Code Inform the operation and maintenance personnel of the impact of the change. Development of your developers' work/ The test environment should be as consistent as possible with the production environment.”
“ The detailed installation steps required by the operation and maintenance department are really annoying!” Zhang Dafang complains.
Bill Say：“ Let's set a short-term goal first, Fully automate deployment. The script deployed is led by Lao Li's operation and maintenance department, If you have a problem, put on weight to help,
In the future, this deployment script will be used in all environments!”
Zhang Dafang and Lao Li nodded again.
Bill Say again：“ The last point is measurement, E.g. percentage of failed deployments, User openedticket Number, Mean time to failure recovery, etc, These old Li should know better than me. We will use these metrics to measureDevOps How's it going,
The most important thing is that no matter what tools we use, Method, If there is no reduction in demand at the end of the day, from proposal to online, Time delivered to users, It was all a failure
. I ask you both to do everything, To reduce this time, Not once, Twice, But continuous, Stable delivery to users.”
Zhang Dapang said：“ thisDevOps That sounds good, But it's hard to implement!”
Bill Say：“ Let's choose a product as a pilot, Then expand the promotion!”
Friends who like this article, Welcome to click on the figure below to pay attention to subscription number programmer Xiaohui, Watch more