EOC Shadow and EOC Buddy Expectations

The engineer on-call (EOC) shadowing process is designed to provide new engineers with practical, hands-on experience in managing live incidents, responding to alerts, and ensuring system stability. Shadowing allows new team members (Shadows) to observe and gradually take on the responsibilities of an EOC under the guidance of an experienced engineer (EOC Buddy).

This document outlines the key expectations for both the EOC Shadow and EOC Buddy, ensuring a structured approach to learning and support throughout the shadowing process. By clearly defining the roles and responsibilities, we aim to ensure the Shadow gains the necessary skills, knowledge, and confidence to handle real-time incidents effectively, and that the EOC Buddy can provide the appropriate guidance and mentorship.

The goal is to create a collaborative and educational environment that prepares the Shadow to independently manage on-call duties while maintaining system reliability and efficient incident response. This document serves as a guideline to support the successful transition of Shadows into full-fledged on-call engineers, ensuring that they are fully equipped to take on the challenges and responsibilities of the role.

Expectations for the EOC Shadow

The following outlines the expectations for the EOC Shadow during their shadowing period. These steps are intended to help the Shadow gain the necessary experience and confidence before they take on full EOC responsibilities:

1. Join the shadow rotation after completing onboarding: The Shadow should enter the shadowing rotation only after completing the initial onboarding tasks for Site Reliability Engineers (SREs) or their equivalent role. This also includes a discussion with their manager to ensure they are fully prepared to begin shadowing and learning from experienced on-call engineers.

2. Dedicate one day a week to shadowing: The Shadow is expected to commit to shadowing at least one day a week over several weeks. This allows for gradual exposure to EOC duties and gives the Shadow time to build their skill set. As they approach their own on-call shifts, their involvement should increase, allowing them to take on more significant tasks with greater confidence.

3. Actively participate in incidents and alerts: During the shadowing period, the Shadow should actively engage in incidents and respond to alerts. This includes following along during investigations, observing how decisions are made, and offering assistance where possible. By being hands-on, the Shadow gains practical experience, which is crucial for their development.

4. Check in with the EOC at the beginning of each shift: At the start of every shift, the Shadow should check in with the EOC. This check-in sets expectations for the shift and allows the Shadow to express their readiness to be included in investigations or incident responses. It helps ensure that both the Shadow and the EOC are on the same page regarding tasks and responsibilities for the shift.

5. Join on-call handover discussions: When possible, the Shadow should participate in on-call handover meetings. These discussions provide important context for ongoing issues, incidents that occurred during the previous shift, and tasks to focus on. Attending these meetings is crucial for the Shadow to understand the continuity of operations and prepares them for independent on-call work.

6. Act as the primary responder during the final shadow shift: In the final shadow shift, the Shadow should take on the role of primary responder, managing alerts and incidents as they come in. The EOC will act as secondary support, available to step in within a 5-minute escalation window if the Shadow encounters challenges. This serves as the final test before the Shadow becomes an official part of the on-call rotation.

7. Engage in post-incident discussions and analysis: The Shadow should actively engage in post-incident reviews, where the team reflects on how alerts were handled and what actions were taken. These reviews provide key learning opportunities and help the Shadow gain insights into how incident responses can be improved or handled more efficiently in the future.

8. Participate in the weekly on-call handover meeting: Additionally, it’s recommended that the SRE shadow participates in the weekly on-call handover meeting within their time zone. This meeting provides an opportunity to review and discuss the incidents from the previous week, which can be invaluable for staying informed and prepared.

9. Participate in a “gameday” exercise as graduation: Before officially joining the EOC rotation, the Shadow should participate in a “gameday” exercise. This is a simulated incident scenario that allows the Shadow to apply their knowledge and skills in a controlled environment. Successfully completing this exercise serves as a form of graduation, signaling that the Shadow is ready to handle real incidents independently. Check for the gamedays calender and if one is planned, with the help of the SRE buddy setup one. Check the Gameday calendar, and if one is already scheduled, you can join it. If none is planned, coordinate with the SRE buddy to organize one.

These expectations ensure that the Shadow has the necessary hands-on experience, support, and training to handle the responsibilities of an EOC effectively.

Expectations of the EOC Buddy

The role of the EOC Buddy, or the experienced EOC, is critical in guiding and mentoring new engineers during their shadowing period. The EOC Buddy plays a key role in ensuring the Shadow gains practical insights, understands real-time incident management, and builds the confidence needed for independent on-call responsibilities.

This section outlines the key expectations for the EOC Buddy, focusing on communication, mentorship, and actively involving the Shadow in day-to-day on-call tasks. By following these guidelines, the EOC Buddy helps foster a supportive and educational environment, ensuring the Shadow’s successful transition into their future role.

1. Awareness of shadowing: It’s important for the EOC Buddy to be aware when they are being shadowed. This should be communicated ahead of time, ensuring that the EOC Buddy knows when someone will be observing and learning from their actions. Proper communication sets the stage for a more productive and collaborative experience.

2. Check in with the Shadow: At the start of each shift, the EOC Buddy should check in with their Shadow. This helps to establish expectations for the shift, including clarifying what tasks or areas the Shadow can assist with. This also ensures that both parties are on the same page from the very beginning.

3. Share thought process during incidents: During the course of an investigation, the EOC Buddy is expected to actively share their thought processes with the Shadow. This includes explaining which dashboards they are monitoring, which runbooks they are following, and the specific commands or actions they are taking. This transparency allows the Shadow to better understand the investigative process and learn in real-time.

4. Keep the Shadow informed: The EOC Buddy should make an effort to tag the Shadow in all relevant incident issues, Slack discussions, and investigations. By doing so, the Shadow remains informed and engaged with ongoing incidents, making it easier for them to follow the case and participate when appropriate.

5. Involve the Shadow in some tasks: The EOC Buddy should encourage the Shadow to take on smaller tasks where appropriate. This could include handling minor alerts or assisting with other requests related to the EOC. These practical tasks provide the Shadow with hands-on experience, which is vital for their learning and development.

6. Offer guidance and clarification: Whenever the Shadow requires clarification or guidance, the EOC Buddy should be available to provide it. Whether it’s explaining a particular alert, answering questions, or giving direction on what steps to take next, the EOC Buddy plays a crucial role in supporting the Shadow’s learning process.

7. Participate in post-shift reviews: After the shift, the EOC Buddy is encouraged to participate in post-shift reviews with the Shadow. These reviews offer a chance to reflect on the alerts handled and actions taken during the shift. It’s an opportunity for the Shadow to ask questions, and for the EOC Buddy to provide feedback, ensuring the experience is as educational as possible.