Click above “ Programmer Xiaohui ”, choice “ Official account ”

Interesting and meaningful articles arrive at the first time !

This article is reproduced from the official account.   Manon turns over

The war of development and operation and maintenance
Five days ago , Zhang Dafeng's development team delivered a batch of new codes to the operation and maintenance department , This is an important upgrade that users have been looking forward to for a long time , The deployment went very well , Everyone is very happy . 

But today's production environment CPU Continuous approach 100%, There are several servers down It's time , Operation and maintenance boss is furious :“ It's the third time ! Zhang Dafang , What happened to your development team ?
New code on line CPU Just 100%!”  

Zhang Dafang is not willing to show weakness :“ We tested it very well in the test environment , User pressure is much greater than production environment , Code is rock solid , You must have configured something wrong !” 

“ impossible , We are deployed in strict accordance with the steps you require , It must be your code !” 

“ Then why is the test environment OK ?”


Two executives have been arguing for a long time , There is no good solution , We stayed up all night , Roll back the code to the previous version , Burn incense and worship Buddha , Hope no more problems . 

After this toss , The bonus at the end of this year is expected to be up in the air ! 

Zhang Dafang feels extremely depressed , It's hard to calm down , Black face came to the tea room and poured a cup of coffee , Sitting there and drinking it again, I feel that the operation and maintenance department is too difficult to cooperate ! 

Look at the new recruits , No business at all , They want to “ Shirk responsibility ”, Often said :“ I don't know the business , Content of this launch , You should write every step clearly , I just do it , Don't ask why , It's not my responsibility that something goes wrong .” You say it's annoying !
Does he want to be a robot ? 

also , Their operation and maintenance team focuses on different people , Someone is responsible for database script execution , Someone's in charge Web The server , Someone's in charge SSO ,
oh my god , I have to pull a bunch of people over for several meetings before I go online , Familiarize them with the operation steps .
This man deployed once , I'm not familiar with it , Next time I change , It's also called human resource pool , Can improve efficiency , But new people need to learn from the leader , How could it not be wrong ?! alas ......
Memories of Zhang Dafang
After two cups of coffee , Zhang Dafang calms down a little , Think about it , The reason is that the software itself is too complex , Not only the development is complex , Deployment is also super complex , Every time you deploy, you peel people off . 

Zhang dapang can't help thinking about the software development and deployment process he has experienced in recent years . 

In College , Do some small projects with the teacher , development , The tests are all done by one person , Deployment to users is also self-made , Almost no trouble . 

Enter a small company after graduation , What we did was C/S Architecture system , With a development team , Test team , The development team wrote the code , Test by test team , After that, you can deploy to the customer . 

Generally speaking, the implementers are also the brothers of development or testing , I also worked part-time , Take the deployment document and CD , Go to the customer and install the system on the customer's machine in strict accordance with the steps , Basically, there's no big problem , Even if there's a problem , It can be solved after debugging on site , Big deal, call the development brothers to stay up late together .

And then the wave of the Internet came , I also switched to this Internet company , Make a car appointment system on the Internet , Provide car appointment service to users , You don't have to go to the customer to install the software , The company operates independently , It's all right to maintain this system . 

But the online car Hailing system is comparable to the original stand-alone software ,C/S Software is much more complex , Especially in the face of massive users' high concurrent access , Various technical problems need to be solved , Great challenge .
The system is not only complex , You also need to 24*7 How to run , Part time developers or testers are no longer able to maintain .

The company has set up operation and maintenance (Operations) department , Responsible for system deployment , Routine maintenance , monitor . Unprecedented improvement in the status of operation and maintenance personnel , of course , The demand for their skills has also increased unprecedentedly . 

Zhang Dafang has seen an email about the operation and maintenance for recruitment : 

Skillful use Linux, unix, windows operating system ; 

Proficient in all kinds of common server software (tomcat, apache, nginx,redis,mysql...) Configuration and optimization of  

Understand the principles of load balancing and high availability , as LVS,Keepalived etc.  

Familiar with network principle ,TCP/IP agreement , Master at least one scripting language . 

Can use various configuration management and deployment management tools . 


in short , Operation and maintenance is almost as important as development . 
The gap between development and operation and maintenance
To speed up delivery , First two years , Led the development team to implement agile transformation , Successfully transformed the original waterfall development mode into a trot , Agile mode of frequent delivery .  

( picture source : 

Through agile software development , Put people in need , Developers , The testers pulled together , Forming the so-called “ Feature team ”, Separate requirements into independent ones , Stories of value to users , Prioritize and develop later , test , It can even deliver several independent requirements every two weeks .

 ( Manon turn note , See 《 Vernacular agile software development

The successful agile transformation is highly recognized by the company , They also exported a lot of training . 

Although it can be delivered frequently , But they can't go online frequently , The reason is simple : The operation and maintenance guys always want the system to be stable , stable , Restability , Stability prevails .
So they don't want to go online frequently , Isn't that making trouble for yourself ? 

It's the opposite of agile software development , Agility is about embracing change !

 ( Development requirements change , Stable operation and maintenance requirements , Image creation from, Lao Liu redraws ) 

I figured out the essential contradiction , Zhang dapang knew that he couldn't solve the problem , It must be solved by the upper level . 

Zhang Dafang goes to find it immediately CTO Bill, I hope he can come up with some good ideas .  
Dev + Operations = DevOps
What he didn't expect was , Lao Li, operation and Maintenance Director, is already in Bill In the Office , Zhang dafangxin says no , Maybe the villain told me first . 

Bill The sight of chubby Cheung , Let him sit down first , Listen to Lao Li “ contradiction ” and “ Warfare ” Finish . 

Lao Li is nagging , It's similar to what I think . 

Bill Say with a smile :“ Big fat , The development process of software is basically the requirement -> development -> test -> deploy , Now your team has put the needs , development , Test to ‘ unite ’ Together ,
It seems you want to ‘ unite ’ A new partner !” 

“ Is it Lao Li's operation and maintenance department ?” 

“ you 're right .” 

“ That's impossible. , Our goals are totally different , One needs to change , One needs stability , How can we play together ?” Big fat, very surprised .

“ No , In the future, we will focus on business objectives , Take the value of users as the only criterion , The assessment and evaluation mechanism of the team should also be changed , The success of both individuals and teams depends on the overall development - Evaluation within the operation and maintenance life cycle , Development completed a lot of user requirements are not necessarily successful , Operation and maintenance guarantee system down It doesn't have to be a success ! Only the functions that users want are implemented in time , Deployed successfully , It's only successful if it's used stably .
” CTO Always good at using MBA To talk . 

“ That is to say, we are required to cooperate closely in operation and development ?”  Lao Li then asked . 

“ Yeah , Now there is a hot word called DevOps, It is to break through the fence between the agile development department and the operation and maintenance department , Form a closed loop .”

 “ Are we going to add another department , call DevOps department ? recruit DevOps engineer ?” 

“ No, no , If we add another department like this , Didn't another wall be added ?
We were supposed to break the gap between development and operation and maintenance teams . In fact, the goal of the operation and maintenance department is not only to make our online car Hailing system more stable and efficient , We need to support the rapid evolution of business —— This is consistent with your goal of agile software development .”

 " But we can't deploy it frequently ? The contradiction between speed and stability can't be solved ." Old Li sighed .

 " I know that Zhang's team is implementing the transformation of microservice , In the future, it's not a big Mac application , It is based on micro services , That would be much easier , Frequent deployment is possible , And it's much easier to roll back when something goes wrong , I don't need you to stay up late !"

Zhang Dafang and Lao Li both nodded in agreement . 

 “ What should I do ?” 

 “ The first is the change of concept , You can't shirk responsibility for each other in the future , Blame each other , And we have to share the responsibility ! Development of a project , deploy , maintain , It's about both of you , Both parties are responsible for the business , What's wrong , You need to work together , Joint solution . You need to give more members when you go back ` Brainwashing `.”

  Zhang Dafang thought , We just worked together to roll back the code .

 “ also ,”,Bill Take a look at Lao Li ,
“ Operation and maintenance personnel should also understand the business ,Code Inform the operation and maintenance personnel of the impact of the change . Development of your developers' work / The test environment should be as consistent as possible with the production environment .” 

“ The detailed installation steps required by the operation and maintenance department are really annoying !” Zhang Dafang complains .

Bill say :“ Let's set a short-term goal first , Fully automate deployment . The script deployed is led by Lao Li's operation and maintenance department , If you have a problem, put on weight to help ,
In the future, this deployment script will be used in all environments !”

  Zhang Dafang and Lao Li nodded again . 

Bill He added :“ The last point is measurement , E.g. percentage of failed deployments , User opened ticket number , Mean time to failure recovery, etc , These old Li should know better than me . We will use these metrics to measure DevOps How's it going ,
The most important thing is that no matter what tools we use , method , If there is no reduction in demand at the end of the day, from proposal to online , Time delivered to users , It was all a failure
. I ask you both to do everything , To reduce this time , Not once , twice , It's continuous , Stable delivery to users .”

Zhang Dafang said :“ this DevOps That sounds good , But it's hard to implement !”

Bill say :“ Let's choose a product as a pilot , Then expand the promotion !”


Friends who like this article , Welcome to click on the figure below to pay attention to subscription number programmer Xiaohui , Watch more