For the previous parts in my series Managing an Exercise Program, please see below:
While conducting an exercise is often the most exciting part of the exercise cycle, evaluation and improvement planning lead us to the real reasons why we conduct exercises – to test plans, policies, and procedures. Evaluation doesn’t just happen. It must be a deliberate, planned activity within the design and conduct of an exercise. There are key activities through the design and development of an exercise which link directly to evaluation – starting with the identification of core capabilities to be tested and exercise objectives to be accomplished. These core capabilities and objectives should lead us directly back to specific plans, policies, and procedures which will be exercised and can help us determine the evaluation methodology and approach. As we further develop our exercise, be it discussion based or operations based, the decisions we make influence evaluation and the resources needed for it.
When exercise planning begins, we identify staff to fill key roles – including exercise director, simcell manager, lead controller, and lead evaluator. The lead evaluator doesn’t necessarily have to be involved in all aspects of exercise design, but they need to be informed of key points and should be reviewing draft materials (i.e. the explan or sitman) to become familiar with the details of the exercise and to help them assemble their portion of the control and evaluation plan. The lead evaluator should ideally be present at exercise planning meetings to become even more familiar with the details, the people, and the facility(s) of exercise play.
Staffing evaluation has a number of options. Often times evaluators will be selected based on matching their individual areas of expertise with the functional areas of the exercise (i.e. fire, EMS, public health, EOC management, shelter operations, etc.). You may have a need for more than one evaluator per functional area, particularly if that function is a major component of the exercise and there is a lot to observe. Remember that if there are multiple exercise venues, they each need to have evaluators assigned who can observe all critical areas. The bottom line is that evaluation should respect the amount of time, effort, and resources invested in the design and conduct of the exercise. If the exercise isn’t fairly and accurately evaluated, it’s simply disrespectful to all that effort and those involved. These concepts apply to both discussion based and operations based exercises. Members of the exercise planning team should be considered as possible evaluators.
Exercise Evaluation Guides (EEGs) give evaluators guidance in their activity. The HSEEP website provides a multitude of EEG templates which you should absolutely modify for your own use. EEGs are provided for each core capability. Just like planning templates (see my rant about the use of planning templates here), these are guides for you to reach success, but can not and should not be used without modification and customization to ensure that they meet your own specific needs. EEGs are crafted hand in hand with exercise design to ensure that all bench marks and expected actions are accounted for. The rating guide on the last page of each EEG is particularly handy to ensure consistency. You may want to consider including a ‘recommendations’ section for each observation as well, which provides an opportunity for you gain insight from your own subject matter experts on how to address a particular area for improvement. This will help when it comes to the After Action Report. I also encourage evaluators to use and ICS form 200 (a blank sheet of paper!) to take some freeform notes. EEGs, no matter how well designed, simply won’t capture everything. Personally, when I evaluate, I’ll refer to the EEGs, but make most of my notes on a blank sheet(s) of paper or a steno pad. I like to mark specific observations, times and timelines, and even inject numbers if I happen to catch them. After the exercise, I then transcribe my notes into the EEG in the appropriate format.
Training of evaluators is important. You may have some evaluators which are new and some that are experienced. Everyone should go through a controller/evaluator briefing prior to the exercise. Those with little experience might require some extra training, which isn’t a bad thing. Be sure that both your C/E plan and the briefing identify clearly what you expect from evaluators, how they should conduct themselves, the measure of allowable interaction with players, how they should take notes, and when you expect those notes to be turned in. The HSEEP website, referenced prior, also has templates for C/E briefings, C/E plans, etc. You’ve likely heard the saying ‘garbage in – garbage out’… well, exercise evaluation is exactly like that. If you don’t invest in good training and job aids (EEGs) for your evaluators, you will not get good evaluations! The Lead Evaluator also needs to lead! They should check in periodically with each evaluator throughout the exercise to see how things are going and give a quick look at their notes and/or EEGs. Good feedback will help your evaluators provide better observations.
Finally, at the end of exercise play, I expect evaluators to help with the hotwash. Sometimes hotwashes are conducted as a single session with all players. In such sessions, the exercise director or lead evaluator should facilitate and evaluators should take notes. If the hotwash is tiered, where each functional area will identify their top strengths and areas for improvement; each functional area is facilitated by its evaluator who should also take notes. This may be followed by a plenary session if possible. Be sure to have your hotwash strategy planned and identified in the C/E plan!
The After Action Report (AAR) should ideally be written by one or two people based on the EEGs and notes of evaluators. It may be possible for some observations to be combined, helping to make the document both more concise and easier to read. Templates are provided on the HSEEP website for AARs and Improvement Plans. Improvement plans are nothing more than a matrix identifying who will be responsible for what improvement actions, who will assist, what the major benchmarks are to improvement, and when it should be accomplished. As with the rest of the AAR, these should be drafted and provided prior to the AAR meeting for comment – of which there is usually a great deal. Once all this is finalized, don’t let this be the last of it! The implementation of improvements is where organizations often fail! Rarely do organizations follow through on improvements, resulting in similar observations being noted in exercises and incidents for years after. The AAR/IP must be championed by someone at an executive level, and coordinated on an ongoing basis by someone who is responsible for tracking progress, coordinating solutions to problems, and reporting progress back to the executive. For most organizations and jurisdictions, I would suggest a quarterly meeting to review improvement planning progress. After a few exercises, you may find yourself addressing improvements of several exercises in one meeting. Accomplishments should be noted, with the opportunity to test the ‘fixes’ in future exercises. All this feeds back into the strategy planning phase of exercise program management – which is where we started in this series.
This is the end of my ‘Managing an Exercise Program’ series. I appreciate all those who have read and provided feedback. Please continue to do so, and best of luck on your next exercise!
I’d be remiss if I didn’t mention that my company – Emergency Preparedness Solutions, LLC - can help you with exercise program planning and management; training and exercise plans; and the design, conduct, and evaluation of exercises; as well as training in HSEEP, exercise design, and exercise evaluation. Give us a call or email and we’ll be happy to discuss what we can do to help your organization’s preparedness.
I just received my April edition (number 627) of the Natural Hazards Observer (NHO), a bi-monthly publication of the Natural Hazards Center of the University of Colorado at Boulder. As of my posting of this article, they have not yet posted the April edition on their website, but they are likely to do so shortly. My desire with this particular post is to encourage a thorough analysis of the topic at hand rather than an extremely narrow perspective as was published in the NHO.
First, I feel a need to mention that The Natural Hazards Observer and I have a tenuous on again, off again relationship. I’ve become rather disenchanted with their articles in the past; occasionally disagreeing completely with authors’ premise and approach (not that everyone is necessarily agreeable to mine). I’ve even written the NHO about the fact that a University center whose goal is to research disasters has presented articles that, to me, come across at times as unprofessional and not well thought out. I’m not really knocking the NHO nor is my intent to pick a fight, rather I’ve been trying to be constructive with them. Given their foundation, I think their readers expect a more scholarly approach. While there is certainly plenty of good information to find in the NHO, some seems questionable. One of their tag lines is ‘Disaster Research News You Can Use’. I often find that their articles have little to do with research, such as the one I cite below, and are really more of a media regurgitation. There are other, more appropriate forums for that – like blogging – rather than a publication that advertises research.
That said, my particular beef with this edition is the first article, titled ‘All’s Not Fair: Illinois Battles Over Being Too Big To Feel FEMA’s Love’. I encourage you to read the article for yourself and form your own impressions. To sum up the general idea of the article, folks in Illinois are questioning FEMA’s Public Assistance (PA) program thresholds for disaster recovery. Specifically, they take issue with these thresholds being based on per-capita amounts. They argue that larger, more populous, states are discriminated against (their words); and further stating that a larger population doesn’t necessarily equate to a state’s ability to self-fund disaster recovery efforts.
Public Assistance (PA), by the way, is focused on recovering public infrastructure (roads, bridges, public buildings, etc.), whereas Individual Assistance (IA) provides direct assistance to individuals to help cover uninsured losses.
When first reading this, two particular issues stood out to me. 1) They don’t mention the fact that FEMA will also examine a county’s per-capita damages for potential declarations, and 2) High populated states quite frequently receive disaster declarations.
The Texas Department of Public Safety has a great document which summarizes the state and county thresholds for 2014. The NHO article makes no mention of the county threshold, which is where most PA declarations are decreed – for a county or counties, rather than the entire state. While the county threshold is higher, it would generally be easier for a localized disaster to be declared (PA) than the entire state, which is mostly unaffected.
As for state declarations, FEMA provides a great summary by state of major declarations (this is the specific number to look at in this case), emergency declarations, and fire management declarations. By scanning down the list, you’ll see that states like Texas, Florida, California, New York, and Louisiana are among the top contenders for declarations. Other large and more populous states, such as Illinois, are not far behind them. This pretty much blows a hole in the premise that larger, more populated states are ‘discriminated against’. Having been in this business for a while, I know that sometimes you win (the request for a declaration) and sometimes you lose. It makes a significant argument for states to have disaster reserve funds.
All that aside; are there issues of fairness in this traditional per-capita assessment of damages within the disaster declaration process? There may certainly be. The assumption of the current process is that a more populous state has a greater tax base from which to draw funds for disaster relief. This assumption, at a glance, makes sense, but may not be the most fair way of determining if a state warrants assistance. The most significant arguments against this seem to be 1) A lower relative mean income as compared to other states translates into a lower tax base, and 2) Is the tax base even a fair measure of need/capability for a state?
An interesting read on this is a GAO report from 2012. It seems to me, a guy who is not a FEMA disaster recovery expert – but I’ve been around the block a few times – that there are some issues of fairness and perhaps better ways to assess a state’s true need for Federal assistance. A fair discussion of this topic, however, requires having a 360 degree perspective of all the issues involved. Perhaps it’s a good topic for a research journal?
After over a year hiatus from my series on Managing an Exercise Program, I’d like to pick up where I left off to discuss Conducting an Exercise. Despite being away from the series for a while, it still seems to have some viewing popularity – much appreciated! If you’d like to reference the first eight parts of the series, links are provided below:
Since writing Part 8, the new HSEEP guidance (April 2013) was published. Thankfully this update didn’t dictate wholesale changes in the process we’ve all come to be familiar with. There were some subtle changes, such as changing ‘planning conferences’ to ‘planning meetings’, and stressing the involvement of elected officials in the exercise planning process. Similarly, I’m thankful that certain changes weren’t made – one of which was the proposed elimination of terminology of ‘discussion based exercises’ and ‘operations based exercises’. These general identifiers have become commonplace amongst exercise designers for decades. Upon having the opportunity to review a draft version of the new HSEEP guidance, I was quite opposed to that change.
Obviously conducting the exercise is the most exciting part of the process. It’s where all of your hard work comes together. That said, this is where it can all fall apart if you haven’t prepared property in the first eight parts and if you don’t pay attention to detail. I’m breaking exercise conduct down into four smaller parts: Set up, briefings, play, debrief. Remember that my intent with these postings is not to be fully comprehensive, but to emphasize areas where I have learned or have seen others struggle. In all facets of emergency management we need to continue a culture of sharing if we are to be successful.
The preparations for your exercise should have been finalized in Part 8 where you addressed logistical requirements. You should know where your exercise venue is, how people will be seated/where they will be working, the location and support for a SimCell (if you’re having one), and even matters such as parking, food, restrooms, and audiovisual equipment. Additionally, set up includes having print material available for all participants. When setting up, consider the flow of people, quite literally from the road to the parking lot, from the parking lot to the facility, into the facility through security, sign in, and to their first location for the exercise (this may be their only destination or it may be a large briefing room). Having conducted exercises in locations not familiar to some participants as well as these exercises being larger than the facility’s parking lot can handle, you must ensure that invitation packages have specific directions on how to get to the parking area and how to get from the parking area to the venue. Be sure that signage is prominently displayed. It may even help to have staff directing people. Be sure to also let them know what manner of identification is required to enter the facility. Again, signage is often necessary to get people from the building entrance to check in location and then to the meeting location. Signage for the refreshment area and restrooms may also be needed.
Have everything set up for people before they arrive. Use table tents, if needed, to help identify seating, and have copies of all necessary materials at their seating location. Materials like SitMans and ExPlans have a lot of content, so it’s nice to have these materials available for the early-comers to being reading through.
Be sure to set up early and test EVERYTHING. Make sure AV equipment and communications equipment are tested.
Depending on the type of exercise you are conducting, there can be a number of briefings needed to take place before the exercise begins. Some of these briefings may need to take place well in advance, even a day or more before the exercise, depending on availability of audiences for each of them.
The new HSEEP guidance includes a specific briefing for elected and appointed officials. This is something to usually conduct well in advance of the actual exercise to outline the agenda/timeline of the exercise and what will be expected of them. They will likely want to interface with any media and should know what to say about the exercise. They may also want to ‘kick off’ the exercise at the player briefing.
Controllers, evaluators, and SimCell personnel should receive a briefing to ensure they are familiar with the exercise and their rolls. They all need to know what is expected of them and of the players, how and when to interact with the players, and what may need to be reported to their respective leaders and/or to the exercise director. Evaluators will need to become familiar with their assigned EEGs and what the expectations are for evaluating. Additional time will need to be spent with SimCell personnel to familiarize them with the equipment being used, how the MSEL is structured, and how injects will be delivered. If any actors are being used, they will also need to be briefed on their rolls and what their expectations are. Ensure that everyone is familiar with the safety word in the event of the necessity to immediately stop the exercise.
If observers are expected at the exercise, have a briefing ready for them. I suggest treating them like VIPs when possible (most of them usually are). Don’t just let them walk around to figure things out for themselves. Give them tour guides who can take them throughout the venue and talk about what is happening. Schedule their arrival when possible, and consider having an elected or appointed official there to greet them. Keep in mind that some observers will be very high tier VIPs. Vigilant Guard exercises are often visited by the head of the National Guard Bureau – a four star General. These visits are fantastic and very much emphasize the importance of the exercise, but they can also be a little disruptive. These high-tier VIPs will often come with their own entourage and/or security. They will want a brief tour and will stop often to shake hands and have pictures taken. They may even want a break in the exercise to speak. Try to be aware of these expectations up front if at all possible. If not, just go with the flow and be flexible.
The media is another form of observer. All media should be scheduled. Be sure to make it worth their while, where they can catch some video/picture footage of interesting activities. Maps and wall displays make for great footage as well. Remember operational security! If something is sensitive or classified, it should not be anywhere where it can be seen. The media should also be provided a tour and an opportunity to get a statement from the elected and/or appointed officials. They may also want to interview players. I generally don’t allow this without preparation of the specific player to ensure that the right messages are delivered. Also, be sure to let players know during the player brief that there will be observers and media coming through and what is expected of them.
Lastly, the player briefing. This briefing will introduce players to the purpose of the exercise, the ground rules, the facility, timeline, controllers and evaluators, and expectations. Players should be briefed on how and when to interact with the SimCell if you have one. It should always be reinforced that the exercise is ‘no fault’, and that they, individually, are not being evaluated, rather it is the plans, policies, and procedures they are using that are being evaluated. Lastly, players are briefed on the scenario. Be sure that the ExPlan/SitMan has the detailed scenario for their reference.
Finally the moment of truth arrives! Remember that the purpose of the exercise is to accomplish the objectives, however, know that you may not accomplish all the objectives in the timeframe you have. Being flexible, and knowing you have to be flexible, are two very important aspects of running an exercise. Regardless of how well we think we’ve written an exercise, the responses to our prompts are entirely up to the players. They may accomplish the objectives faster or slower than you expected. Likewise, they may struggle a bit. This is where good exercise control comes in. Controllers/facilitators should observe the tempo of the exercise. Is everyone engaged? Are they needlessly overwhelmed and frustrated? Are they bored? Are they not following through on activities? All these observations should be reported back to the exercise director and SimCell so the tempo can be moderated.
Be sure to have contingencies in the event that players do or request the unexpected. Additional injects that are held back are always a good idea. You may need them to prompt certain activities in the event that players do not take the initiative to do so. Players may also ask questions or make requests of the SimCell that aren’t expected. While we can’t anticipate every need, we can be prepared for them. Have copies of plans, policies, procedures, and maps available to the SimCell. They may have to take some time to research and come up with an answer then get back to the individual. The SimCell manager should be smart when situations like this arise, however. They should always consider if a potentially inaccurate but reasonable answer is acceptable or if the answer must be completely accurate… or even if the answer is required for exercise play at all. There is no sense having SimCell personnel research an answer if the answer doesn’t much matter within the scope of the exercise.
Multi-day exercises may require the exercise staff to meet at the end of every day for a mini hotwash and to evaluate how the exercise is progressing. Where are the players in respect to where you expected them to be in your design of the exercise? Do you need to retool anything in the MSEL?
On large exercises, it helps for the exercise director to have an aide-de-camp, or assistant. Just like any individual in charge of a large operation, the exercise director is having their attention pulled in many directions and may simply not have the time to address everything, especially in large or multi-venue exercises. Having an assistant is also a great way to train up and coming exercise staff. Certainly consider the use of portable radios to help facilitate communication as well. The exercise director will spend most of their time managing and trouble shooting. They may also have to address some VIP concerns. Be sure to walk around to get a good sense for the exercise as a whole. Are evaluators properly positioned? Are controllers present and visible? Is the SimCell responding to questions adequately? Are players engaged and challenged? Perhaps most importantly, is the boss (the elected/appointed official) happy?
Evaluation is such an important aspect of exercise that I will cover it in its own section – Part 10.
Two significant debriefing activities should take place immediately following the exercise: 1) a player hotwash, 2) an exercise staff debriefing. Players should be led through a facilitated hotwash, reviewing the objectives of the exercise, with evaluators capturing their responses to if/how the objectives were accomplished. Encourage and capture responses both in the positive and the negative: i.e. what did we do well and what do we need to improve upon?
After the player hotwash, a similar process should take place with all exercise staff – controllers, evaluators, and SimCell. I like to not only capture their impressions of if/how objectives were accomplished, but I also like to discuss the conduct of the exercise, again capturing what went well and what needed to be improved upon.
Back in October of 2013, in an article in Emergency Management Magazine, David Silverberg discussed the issue of elected officials rarely being educated about emergencies. Just last month, David Maack published an article in Emergency Management Magazine about how elected officials can prepare for and respond to emergencies. Silverberg’s article not only identified the issue, but also pointed out strategies used in some places around the country to help orient elected officials to the world of emergency management.
The premise of both articles is that this familiarization is obviously lacking, and it’s quite imperative that elected officials are familiar with the concepts of emergency management; the hazards, capabilities, and plans of their jurisdiction; gaps and needed improvements; how to support their emergency manager; and state laws governing emergency management where they are. I’m a firm believer that elected officials do, in fact, need to support their emergency management programs and their emergency manager. Not only on a day to day basis, but certainly during a disaster. The emergency manager is a subject matter expert, and while they take direction from the elected official, the elected official needs to take cues from the emergency manager. The emergency manager needs to be very clear with their boss about what expectations they have, as well. Maack’s article includes a great list of steps that emergency managers should reference when giving an orientation to their elected officials. Obviously it all needs to be put in the proper context for each jurisdiction.
Here in New York State, the regional office of the State’s Office of Emergency Management works with the county/local emergency manager to conduct a Public Officials’ Conference. These sessions, usually conducted in an evening, are intended to not only familiarize elected and appointed officials of a jurisdiction, but to also communicate what is expected of them and to strengthen the role of the emergency manager as a coordinator and subject matter expert. While there are some usual content areas in these briefings, such as NIMS, a briefing on the local emergency plan, etc., it is flexible based on the needs identified by the emergency manager. These sessions, through the years of conducting them, are generally very well received. One of the most important messages is the flow of emergency management. People need to know how the system works and who is responsible for what.
Several years back, when teaching an ICS Basic (I-200) course, one of my participants introduced himself as “the town supervisor – you know, the one who would be the incident commander during an incident”. I made certain, during the course, to point out that elected officials are generally not the incident commander, rather they are the incident commander’s boss. I then spent some time pointing out what the roles and responsibilities were of this ‘agency administrator’.
What thoughts do you have on familiarizing elected officials with emergency management?
Great post on situational awareness with many applications for emergency management.
Originally posted on Steelhenge Crisis Thinking:
This blog is the second in a series that looks at the challenges of managing information in a crisis and how to ensure the top team gets the information it needs. The first looked at “Managing the Upward Flow of Information in a Crisis – What Matters Most?” Here managing information to build situational awareness is under the spotlight – how to pull together that cohesive and informative picture that gives the boss just what he needs and no more.
It is a fact that almost all crisis teams find information management one of the greatest challenges in responding to an incident. Why does this matter? It matters because effective information management is the bedrock that allows the critical decision-making by the strategic crisis management team that will lead an organisation out of a crisis.
Unlocking the Challenge of Information Management
The key steps to unlock the…
View original 674 more words
Great infographic from Entrepreneur.com!
Originally posted on Recovery Diva:
This makes a nice companion to the infographic on business continuitythat I cited last fall.
While I’m amazed by the number of jurisdictions and businesses that don’t have any emergency plans at all, I’m almost nearly amazed by those who only have isolated, single hazard plans. I’ve seen plenty of entities with only fire plans or flood plans.
How does this happen? Many small towns may ask their local fire chief to write an emergency plan for the town – this plan often times outlines the fire response structure but ignores other hazards. Sometimes this myopic view of planning takes place because of a focus or near panic over one particular hazard. These situations are generally not the fault of the people who wrote them. Just because someone is in public safety doesn’t mean they know every facet of it. Emergency planning is as much of a niche as pump operations in the fire service or special tactics in law enforcement. It requires a certain set of knowledge, skills, and experience to be successful and effective.
While it’s logical that a larger jurisdiction will have a more complex emergency plan than a smaller jurisdiction, there are still foundational elements that are common to all. Perhaps the most critical of these is the necessity to plan for all hazards. All hazards planning, if you’re not familiar with the concept, is a process beginning with the identification of the hazards which can impact a community, the impacts associated with each hazard (also known as vulnerability), and the probability of each hazard impacting the community (also known as risk). This is called a Hazard Vulnerability Analysis (HVA). By rating the vulnerability and risk factors for each hazard, a fairly simple algorithm can be used to provide an overall ranking for each hazard. Additional information, such as the availability of resources to respond to each hazard can be included as well.
Obviously there is a lot of context that needs to go into such an assessment, which is best conducted by a planning team to gain the most amount of input. Comprehensive Preparedness Guide (CPG) 101, published by FEMA/DHS, provides much more information on the planning process as a whole. Historical disaster information should be referenced, but not to the extent of outweighing other inputs. Fluctuations over time in population, topography and development, and climate change are all leading to, as the investment world states it, historical information not necessarily being indicative of future performance.
Scientific data should also be examined, such as flood maps and wild fire risk. Communities should be sure to reference information on hazardous materials, which is required reporting by industry per the Emergency Planning and Community Right to Know Act of 1986 (also known as SARA Title III). Once you have identified and ranked all the hazards, you know what ones to focus on. Perhaps it is the top 8, or maybe the top 12 that indicate a high enough vulnerability and/or high enough risk to incorporate into your planning efforts. Once these are identified, you continue through the CPG 101 planning process to create a comprehensive plan. A comprehensive plan incorporates all elements needed to address the responses to these incidents. Given that the majority of responses are largely similar, regardless of the hazard, one concept of operations will address this in the plan. Separate plans, written as annexes or appendices, then can be created for hazards with very specific issues, such as a pandemic, or to address specific functions, such as evacuation and sheltering.
Is it possible to anticipate every disaster that could possible impact your community? Can your community possibly predict being trampled by a giant man of marshmallow or having busses tossed aside by fire breathing dinosaurs from Japan? No? Then don’t spend much time on these things. Be realistic. That said, a Comprehensive Emergency Management Plan (CEMP) – which is a plan that considers all hazards, all resources, and the whole community – can even help account for hazards you didn’t identify. Do you think Chelyabinsk, Russia had a plan for meteor strikes? Not likely. However, consider how the components of a CEMP could have addressed critical areas of response: mass casualty, search and rescue, debris management, sheltering.
I recall a number of years ago, prior to the 2013 occurrence in Chelyabinsk, where there was a bit of media frenzy over near earth objects (NEOs) such as meteors. The public information officer (PIO) for the state’s emergency management agency received a call from a media outlet inquiring if the state had a plan for meteor strike. The PIO answered ‘yes’, qualifying that with a statement about the value of comprehensive planning. Was there a specific plan for meteor strikes? No. The likelihood is so slim that it’s not worth putting effort toward (although I’m sure someone has). Does the CEMP cover all the necessary response components? Absolutely.
Author’s note: Unfortunately I don’t recall where I got the picture from, so I’m unable to give proper credit. If anyone knows or claims it, I’ll gladly give credit or pull it down.