［Engineer Interview］The Fun and Value of Development Support / SRE (as Related to a Development Engineer)

 

Professional Backgrounds and Motives for Joining the Company

All right then, should we get started? I’m nervous; actually, I get really nervous easily. Well, I am Morita with the Development Department. I would like to know about your (professional) backgrounds and what brought you to join LINE. Can we begin with you, Mr. Matsuzaki?

Sure, I’m Matsuzaki. I was doing development operation of a working system in Fukuoka before I joined LINE Fukuoka. Well actually, I was kind of like a jack-of-all-trades. Since being a member of livedoor, I had been referring to the information provided by other engineers and integrating it into my duties; that was what got me interested, and joined LINE Fukuoka. It has been exactly two years since I entered the company, but since there was no Development Support Desk at that time, I started out as a Development Engineer in the beginning. Then I got involved as an establishing member of this team a year ago, which leads us to today.

I guess a jack-of-all-trades is like so called a Full-stack Engineer, but did you have any specific reason for choosing development support rather than service development?

Well, it was because there were engineers who I look up to on the team, so I wanted to work with them. Also, I participated in something called the “Iikanjini Speed Up Contest” (ISUCON), but didn’t come close to beating my peers, so I started to think that if I worked in the same team with them, I might be able to gain something from it.

Thank you very much. How about you, Mr. Nakamura?

After six years of experience at a small service company, I transferred to a Web service company. I had two motives for joining LINE Fukuoka:  one of them was that I had heard Mr. Wada (LINE Fukuoka’s Development Department Head) speak while still at my previous job, and found it interesting. The other was that I often heard that there were amazing engineers here.

Just like Mr. Matsuzaki, I joined the company as a Development Engineer at first, but I had experience doing just about everything from server configuration on. So when I heard they were making a new team which would be doing things more closely related to infrastructure, I got interested in it and joined.

You said you were in Web service company; were you doing service development, too?

Yes, I was mainly developing a shopping cart system for the company’s service. I also developed live chat services, browser games, and so on.

That sounds interesting. So what exactly made you want to go into Support Development?

I had had many chances to do work more closely related to infrastructure (e.g. server configuration) and I thought it would be fun to do it as my primary job, so I volunteered.

I see, thank you very much.

Actually, I worked with Mr. Matsuzaki and Mr. Nakamura doing server-side development for the ” LINE Fortune” project that I’m now the lead of. As Mr. Matsuzaki said, there was an open call to establish a Development Support Team here in Fukuoka at that time, but no one expected that two members from our team would volunteer for it. It was tough losing both of you!  (All laugh)

Ok, how about you, Paul?

I’m from the United States and graduated from North Carolina State University. After that, I moved to Silicon Valley and worked with startup companies for seven years. At the first company, I gained a little experience doing infrastructure work, but I started doing infrastructure (DevOps) almost exclusively at the second company and found it very interesting and enjoyable.

At that time I was taking Japanese language lessons as a hobby, and I really enjoyed them, so when it came to deciding the next step in my next career, I started thinking about studying a lot of Japanese and living in Japan. That’s when I learned about LINE Fukuoka, and I joined the Development Support Team where I can make full use of my experience on the operations side.

Why did you choose Fukuoka instead of Tokyo or some other city?

Personally, I thought that Tokyo is a good city for traveling but not for living in. I had been to Fukuoka before, loved the good food and nice people here, and I always wanted to live in Fukuoka.  I’m convinced that I made the best choice.

Your Japanese is great but did you study by yourself?

No, it’s still improving, but I am fine with daily conversations. When I was living in San Francisco, I often met with Japanese friends and spoke half in Japanese and half in English. I think that helped me to acquire conversational skills.

I see. Do you feel that there are any differences between working in Japan and in America, such as the people or environment?

Yes, there are some, but not many since I was in an IT company. If I had worked for other kinds of businesses, there would be many more differences. For example, there are no strict rules about things like using honorific language in this business. (Laughs)

So you feel comfortable with the business and company culture.

Major Missions are to Improve Tools and Docker Design Using Prometheus /Grafana

I’d like to ask you about the Development Support Team to which you all belong. I know that the Tokyo office has the same kind of team, but could you roughly tell me about your role in the company?

Sure. One of the major missions of the Development Support Team at LINE Fukuoka is making improvements to monitoring. More specifically, we’re improving unique in-house tools used from the livedoor days like Kurado, as well as the hard-to-use parts of monitoring tools that have been used for a long time. We make them easier and better by using Prometheus and Grafana.

There has also been talk about wanting to use containers such as Docker, which are used in some projects at LINE (Tokyo) but not yet at LINE Fukuoka. At the In-House Hackathon* held at the end of last year, we produced a something called “Bot as a Service” (a platform that lets LINE Bot developers easily operate Bots). With the knowledge we gained from this experience, we’re redesigning it for implementation with not only Bots but also general Web apps using Docker and Kubernetes.

*: For more details, see   http://linefukuoka.blog.jp/archives/68669487.html

I see. So you said that you discussed and decided to try something new as a team, but how do you get new tasks? Based on what you said, it sounds like you communicate by proposing something interesting on your own, and if another team member says, “Okay, let’s do it!”, then that’s what you do?

Well, we have a very wide range of duties and a lot or jobs to do. Of course there are some things that we have to work on immediately, but we believe that having long-term and short-term views are necessary to our job. We try to have foresight and think about what is really needed right now, and then prioritize each job.

Are you prioritizing Docker, which you mentioned earlier, or the lately hot topic of DevOps?

In terms of Docker, we’ve talked about improving the operating environment. When we integrate a server into a service, we do it in a way that will run an application. It would take a long time to explain in detail, so in short, we’re talking about making improvements that would facilitate the simple creation of an app-operating environment by defining the containers that run applications. In a broad sense, this is DevOps.

I ask the Development Support team to prepare servers when I need them during the course of developing services, but do you actually physically handle servers?

No, we don’t. (All laugh) 

There is a dedicated section which does data center operations and infrastructure development at LINE (Tokyo) that handles all of that. People generally referred to as Infrastructure Engineer belongs to that section, whereas we in the Development Support Team handle middleware and the application side. The Service Development section develops the server side and applications.

So the Development Support Team is totally different from the Infrastructure Team.

Yes, we don’t touch anything related to the network, HW setting, or things like that.

I see, thanks for the clarification.

Paul, you said your previous job had to do with DevOps, but are you currently doing some of the same things in the Development Support Team? How do your previous job and your current job differ?

The tools that we use are different, but the jobs are quite similar. At my previous job we were using Salt, and now I’m using Ansible, so they’re almost the same.

Right now I’m trying to create something that will improve monitoring and be easier for developers to use.

Paul, are you using Prometheus for the monitoring part of your work?

Yes, Prometheus。

Prometheus and Grafana are being used in LINE Fortune, too, but I was really interested in the post-configuration numbers. I think a lot of people are paying attention to them.

There were times when I thought the server load seemed heavier, but actually, the reasons for that turned out to be things like campaigns started by the planners. Everyone is interested in how the numbers fluctuate, but since we’re doing service development, monitoring isn’t prioritized, so I’m sure one of the roles of Development Support Engineers is also to support that part of the job.

I think you also teach us about a lot points we’re not clear on. For example, lots of things are displayed in Grafana, but we didn’t know how to view them at first, and you helped us out. In that sense - and this might be speaking strictly in terms of definition - you also deal with SRE work.

Right. When the load increases and we receive an alert, we investigate the cause with the lead engineer of the service and improve its performance. We look at application codes, then write and deliver performance improving patches for the OSS being used.

By adopting Prometheus, we’re now easily able to acquire metrics that we couldn’t before; that lets us raise alerts, so responding to bugs that pop up and investigating their causes has gotten easier.

So you look at things like whether there are enough resources such as CPU and memory, whether there are enough servers to begin with, and whether configuration values like the number of threads are in order. I’m sure the service operation-side really appreciates you looking into all of this.
 

Tasks for Engineers Include Creating an Environment Conducive to Development, the Development of Various Tools for Streamlining, and Operations

I’m sure it’s different for each of you, but what kind of work do you do specifically? We may have touched on it a little earlier, but what about you, Matsuzaki?

Well, the Development Support Team does quite a lot!  (All laugh)

Recently we’ve been involved in planning the on-boarding orientation geared towards new engineers and creating content used in a more hands-on kind of way.

I see. What else do you do?

Well, I talked about this earlier, but we’re using Docker and Kubernetes to create a platform for running applications  (for example, something that would deploy as a Docker container just by specifying a GitHub repository and clicking a button).

Nice! How about you, Nakamura?

I respond to alerts raised by services, and transfer services running on outdated servers to new ones. I also install a dedicated monitoring application called “exporter” that uses Prometheus  (which I know has already been brought up a few times) on each server. I thought it would be cool if we could automate this work, so now I’m concentrating my energy on developing an automation tool. Finally I’m doing development that replaces Promgen with Python.

I get the sense you’re really contributing to OSS.
 

The mission of the Development Support Team is to streamline the development environment and make it conducive to development for the developers themselves. That’s why we updated the NoPaste used in-house to be capable of utilizing Markdown notation, and also created a lossless compression tool that everyone can use  (since implementing lossless compression for images on a per-service basis would be wasteful).

NoPaste is a tool for sharing text over a Web browser, isn’t it? That means not only developers but also members of the QA team are using it; that’s amazing! Thank you.

What about you, Paul, what kind of work are you doing?

My main jobs are related to monitoring, so the development of Promgen and operation of Prometheus. I read about Prometheus BTS and GitHub issues, and even attended the  Prometheus Casual Talks held in Tokyo last year to give a presentation. Almost all of my work is related to Prometheus.
 

People who Like Fine-Tuning / Performance Improvement are a Perfect Fit!

What is the most appealing aspect of the Development Support Team?

I’m an expat, so I’m still studying Japanese, but I like the idea of “improvement”. I think the cycle of making small improvements rather than big ones is fun.

I test new tools and technology. I’m sure service developers could do this, too, but I think it can get complex when you’re doing it as your job, so for me, that’s the most appealing part.

In actual service development, outdated technology is inevitably used a majority of the time, but the Development Support Team gets to try out and learn about new tech all the time, so it’s intellectually stimulating.

So you get to take on challenges. Conversely, is there  (or has there been) anything you’ve struggled with?

Since we mostly deal with new technology, there’s a lot to remember. Some of it may not be documented, or we may get tripped up about something that no one else has before, so we can’t search for a solution. That part of it is tough but also fun.

It sounds like what you find interesting differs from what a Development Engineer would find interesting. In all of your opinions, what kind of people are suited to Development Support work?

People that love fine-tuning!  (All laugh) I think people who like improving performance would be suited to the job.

I think we’re basically the same as other developers, but people who like to continually learn about new technology and acquire knowledge are particularly well suited to this kind of work. We always have something new in front of us, and when we don’t understand something, we look into it, which gives us a new experience.
 

Changing Knowledge and Perspective: Takeaways from and Praise for Development Support

What’s the best part about being a member of the Development Support Team?

When engineers show their gratitude for our efforts. When we’ve researched something and resolved an issue, it feels really good to hear the engineers say that they appreciate us helping them to understand something they didn’t know about.

For me, it’s understanding how infrastructure and middleware are constructed / configured, and how they operate. Since LINE is a large organization, various teams are experts in a certain type of technology, but since I began development work on “LINE Fortune” right after I joined the company, I didn’t know that, and everything seemed unclear.

After I transferred to the Development Support Team, I had more opportunities to communicate with expert teams, and came to understand aspects that I didn’t get when I was doing service development.

I see. Actually, I also consult Development Support when I don’t understand something that the   expert database engineers have told me. For example, I know that when a database server breaks then a fail over occurs, but I didn’t know how it operated, and I was able to learn more about unseen parts like that.
 

You can also learn about the differences between each service. In talking with Service Development members, it’s fun to learn about how a particular service is handled. Depending on its characteristics, the middleware selected for a given service may also change.

You’re right; I don’t actually know what the Service Developer sitting next to me is doing, but I’m curious to find out.

My final question is, what does the Development Support Team hope to do in the future?

I’d like to appeal the existence of the Development Support Team even more strongly. We are in a position that is indirectly related to services:  our main job is not to develop them ourselves,  but rather to get involved in a portion of the work that allows Service Developers to concentrate on development, as well as  (like I mentioned earlier) to make improvements.

There is still a lot that we can do, and a lot that we want to do. If each of us can make achievements as engineers, then I think we’ll be valued as a team.

I agree. Even though we are all engineers, I now know that what we feel and our perspectives will differ depending on our position. 

Thank you for your time and insight today!
 

 

LINE Fukuoka is always accepting applications from Engineers.

Please confirm job openings and related information at the links below.

・Development Support Engineers

・Other Positions

・In- house Hackathon Report

【JP】  http: //linefukuoka.blog.jp/archives/68424659.html

【EN】  http: //linefukuoka.blog.jp/archives/68669487.html